Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

As far as i know, it was added to increase the portability of xtool.

XTool is going independent, abandoning its helpers, lol.
Maybe in the future it will get its own final compression with high compression gain and higher performance than lolz... It would be great!

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor :)

@Razor thanks for the continued updates!

Hello.
No. It's just a creative and really a gifted step, not more.:)

Hello.
Exactly like that! Like a Precomp compression with the key -c.:)

Update available

Changes

- added feature to inject libraries to main executable

Notes

You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage
More notes

Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

Thank You So Much, Razor12911!
I think this is the really great solution.:)

Hello.
1. Perhaps, just replace old to new.
For example:
If I have xtool.exe with integrated libraries:
xtool.exe inject _libraries\fast-lzma2.dll
xtool.exe inject _libraries\preflate_dll.dll
xtool.exe inject _libraries\zlibwapi.dll

And I want to replace only one a new version fast-lzma2.dll, then:
xtool.exe inject _libraries\fast-lzma2.dll

Or probably, every time there is a new version of xtool, you need to create a new integration.

3. Yes.

I have found something that seem like a bug, and I am not sure if its FA or xtool, but likely an xtool issue.

Say you precomp a file with reflate, but you have an xtool.ini config that contain all kind of codecs, like xmemcompress, quickbms based and so on. But you only used reflate.

Now to decode successfully, you have to have all those exe files + untouched xtool.ini otherwise xtool will throw general error! Even if you don't need any of those for decompression.
That just happened to me. I had to copy all those pointless exe's and exact same xtool.ini for it to work. If for example I edited out those unneeded codecs from xtool.ini and/or deleted crilayla.exe or xmemcompress.exe, then xtool won't decode my reflate pack!

EDIT: No wonder it fails:
Attachment 32214
Why it have to save all those things if it's not needed?

Xtool stores all possible methods because xtool itself doesn't know if are or will be used if all before even precompression begins.

1) yes
2) possibly, in the main post there is actually link to the older releases I just removed 0.3.21 to make people not ask what is the different between this version and the recent update.
3) yes
4) imperfect streams use xdelta for patching, this function initially came from a dll but I separated all dlls from the main executable and gave the options to the user to include such a feature.

reflate issues: some more info
 

Greetings Razor.
Back on page 34 where I wrote about reflate problems, causing crc errors... This time I was able to catch in on console:

Attachment 32246

But the thing is, exact same run may pass successfully on second or more tries(and then maybe fail again, so random dice per run). I am starting to be cautious about HW issue possibility on my side, but then I had no such issues with zlib yet. Using -t8.

Looks like either thread race issue, or my HW memory. This is on xtool v0.5.3. I will investigate this further.

EDIT:
Latest v0.6.2 == same issue.
If xtool do not fail during compression(precomp), then it will always recompress successfully during unpacking - i.e. probably not HW issue after all.
Last stable version without this bug that I have is 0.3.21. Cannot replicate on it. I believe this one is stable.[Its not, later I found..] Unfortunately it miss a lot of streams that newer xtool-reflate can see.
EDIT2:
Forgot to add. Changing chunk size(or not using depth) can help sometimes, but on big enough data stream one chunk size may pass at one point and fail at different one(where again some other size could work and so on). Also, I started having these issues only recently which correspond with me upgrading to later xtool version than I was stuck with.
Also, I think this have way more likelihood of happening on files that do not contain(or very little) deflate chunks. When FA reported CRC fails it was pretty much files that happen to share same extension with those that do.
EDIT3:
Yup, not my HW. I just threw whole Halo Master Collection directory to xtool 0.3.21(without filter) and it completed successfully. Then I tried both v0.5.3 and v0.6.2 and it failed on address violation.
EDIT4:
I will further confirm it later here, but using -lm *may* work as a workaround for reflate. Not due to a memory shortage, but the way chunks/data and threads are handled. No, it didn't help. [ADD: neither did -t1]
EDIT5:
Preflate works without issues, so I had to settle with it. Final repack size is ~1gb bigger though. I do hope reflate get fixed in the future as I prefer it.

samples?

Unfortunately this wouldn't be possible, it need to be tested on whole Halo Master Collection directory. Because you are unlikely to trigger on small data. Even different arrangement of files(sorting) or chunk size can affect where or if the issue happen at all. Closest I was able to get for smaller data sample were 3 ipak files which is still ~1gb and less reliable to replicate. This could be a rare byte sequence that cause this. And if so I don't know if it's within reflate dll or xtool.
Best way to test is to run it on whole Halo MCC directory without any filter(all files), even better with FA gui instead cmd. That is the closest I know.

Is there a list of every valid codec/method that XTool supports? I'm talking about the ones outside of the codecs listed in the included documentation. It seems like a silly question but I havent kept that close an eye on XTool's development and with the dropping of GrittiBanzli support just thought I'd see if anyone has an up-to date list.

Hello Razor12911
Dying Light 2 Stay Human compression XTool 2020 (Database you can provide support ?
Dying Light 2 Stay Human example Data
https://lifeboxtransfer.com/s/5d1043...d-ee4116099aea

-mpng (have to be used with -d1 parameter and reflate/preflate afterwards)
-mflac
-mbrunsli
-mjojpeg
-mpackjpg

Did they change compression? I'm sure in some older version of this game they used just zlib or sort of if I remember correctly.

No changes. ZLIB only in 2 *.pak files. RPACK is no compression

External Codec Error
 

Hello, it's been ages since i had an error !!

This Came directly from Xtool documentation
Xtool version used 6.2

this is my wav.ini
[Stream1]
Name=wav
Codec=wavpack
BigEndian=0
Signature=0x46464952
Structure=Signature(4),FileSize(4),FileType(4),Str eam
StreamOffset=-12
CompressedSize=FileSize + 8
DecompressedSize=0
Condition1=FileType = 0x45564157
Condition2=FileSize >= 4096

this is my xtool.ini
[wavpack]
Encode=wavpack.exe -hh -x4 <filein>.wav <fileout>.wv
Decode=wvunpack.exe <filein>.wv <fileout>.wav

i have been unable to make this even detect the sample FLAC internal codec works and so any other
NOTE: i use this as standalone and not in FA for testing purposes

xtool.ini:

[wavpack]
Encode=wavpack.exe -hh -x4 <filein> <fileout>
Decode=wvunpack.exe <filein> <fileout>

or

[wavpack]
Encode=wavpack.exe -hh -x4 - - <stdin> <stdout>
Decode=wvunpack.exe - - <stdin> <stdout>

i Spoke with Zee :D

it's a bug confirmed, it's been fixed already

Update available

Changes

- added universal lz4f scanner
- fixed issues with database feature
- fixed issues with executable plugin support
- updated lzo codecs

Update available

Changes

- fixed issues with lzo2a and lzo1c codecs

XTool v 0.6.4

Update available

Changes

- updated oodle scanner
- remove xdelta support from oodle and lzo codecs (crc mismatch often generates large diff files)

Notes

I've been getting reports of xtool taking a long time to process oodle streams (or getting stuck) so I reworked the scanner. I reverse engineered the code of oo2rec (I lost the source code) and rewrote parts of the code so xtool should produce better results while being faster.

There is a parameter rework as well, some people may know of the "n#" parameter which when set increases the chances of xtool capturing a stream. The default value now is 32, can be increased to 64, 128... Up to you, there is no limit just keep in mind that increasing this value also means longer precompression times.

Results on DOOM Eternal's "gameresources_4_1.streamdb"

oo2reck
xtool 0.4.7-0.6.4
Rumour has it that it's still processing as we speak (stuck on 5.7%) :rolleyes:

xtool 0.6.5 (-mkraken:l6:n128)

it's fixed :P , adding one big file and all libraries inside along with all versions of xtool isnt a good idea

confiused . .

xt 065 , ( same result with other version )

https://blogger.googleusercontent.co...147/frusta.png

Hi guys!

I need a little help....
Something is wrong here. Tried to precomp a file with deduplication ,but XTool 0.6.5 doesnt make it. It looks like it ignores the corresponding parameter,even .bin file not created! V0.3.1 works fine!
What am i do wrong?


[External compressor:x065-dedup]
header = 0
packcmd = xtool065\xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup=chor065.bin - - <stdin> <stdout>

[External compressor:x065-nodedup]
header = 0
packcmd = xtool065\xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>

[External compressor:x031-dedup]
header = 0
packcmd = xtool031\xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup=chor031.bin - - <stdin> <stdout>

[External compressor:x031-nodedup]
header = 0
packcmd = xtool031\xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>

Results :

Xtool 0.3.1 ,deduplicated : 40.2 GB
Xtool 0.3.1 ,not deduplicated : 42.9 GB

Xtool 0.6.5 ,deduplicated : 42.9 GB
Xtool 0.6.5 ,not deduplicated : 42.9 GB

All processes finished without error. V0.6.5 is the "pure" version ,directly from Zee's ZIP-file...

Btw where to insert --verbose command?
Only one place is accepted in my experiments ,but no verbose infos are displayed...

Only use --verbose for testing. It limits XTool to 1 chunk processed at once. Add it before input / output. You shouldn't combine stdin/stdout with --verbose, use $$arcdatafile$$.tmp/$$arcpackedfile$$.tmp instead.

You can make a batch file near xtool.exe and drag&drop a file onto it. This way you don't use freearc for it.

Example

Nobody ? :(

Found. Damn...
Parameters were changed...

Quick question, or I guess just asking for some clarification, in regards to "bms2xtl", can you only process a given file with your generated database, if the codec is one that xtool supports? So for example xtool wouldn't be able to process a COMTYPE if it isn't deflate, lz4(hc/x/f)... etc.? I guess what I'm asking in essence is am I correct in saying external, quickbms-native COMTYPEs aren't supported.

Use xtool.ini to add COMTYPEs that QuickBMS supports. Bear in mind that QuickBMS built in compression can vary wildly from that of what the original input file is compressed with so this is really only suitable for encryption algorithms where you're more likely to get a correct output.

Also, not every algorithm that QBMS can decompress it can also compress - such as LZ2K, of which no compression code exists publicly.

You also have totally different implementations of algos althogether such as LZSS where it's basically different every time you see it.

In short xtool.ini compression with QuickBMS is hit/miss.

Hi!

How can i pass the --dedup parameter into the -m chain ?

Just put --dedup into XTool packcmd in arc.ini.

Currently using that method. Is it possible to pass --dedup ?!

Just delete --dedup