XTool 2020 (Main Project)

[CR2032]

Originally Posted by Razor12911

Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

Thank You So Much For Your Hard Work, Razor12911!

[Cesar82]

Originally Posted by Razor12911

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -

XTool is going independent, abandoning its helpers, lol.
Maybe in the future it will get its own final compression with high compression gain and higher performance than lolz... It would be great!

[L33THAK0R]

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

[Masquerade]

Originally Posted by L33THAK0R

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor

@Razor thanks for the continued updates!

[CR2032]

Originally Posted by Masquerade

No, it's just a final compresison option. A bit like how precomp has built in LZMA compressor

@Razor thanks for the continued updates!

Hello.
Exactly like that! Like a Precomp compression with the key -c.

[CR2032]

Originally Posted by L33THAK0R

Are there any current situations/titles that could benefit from the fast lzma2 compression option? Not fully awake, apologies if this is a tad silly to ask, with the answer being plainly obvious.

Hello.
No. It's just a creative and really a gifted step, not more.

[:( Sad8669]

As far as i know, it was added to increase the portability of xtool.

Notes:
I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

[Razor12911]

Update available

Changes

- added feature to inject libraries to main executable

Notes

You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage

xtool.exe inject dll_file

More notes

Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.

[CR2032]

Originally Posted by Razor12911

Update available

Changes

- added feature to inject libraries to main executable

Notes

You may notice that the libraries folder is getting filled with a lot of dll files which xtool uses so reduce this cumbersomeness you might want to embed all of these dlls within the main executable and placing the dlls near xtool.exe is no longer needed as they will become part of the executable.

This feature is added to promote portable mode where all you have is the files you want to process and xtool.exe with no libraries nearby.

Usage

xtool.exe inject dll_file

More notes

Only inject lz4, zstd and oodle when you are sure that your input will never need library swaps as these libraries depending on version determine precompression ratio. zlib, reflate and some other libraries do not as every version produces the same results.

Thank You So Much, Razor12911!
I think this is the really great solution.

[Cesar82]

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

[CR2032]

Originally Posted by Cesar82

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

Hello.
1. Perhaps, just replace old to new.
For example:
If I have xtool.exe with integrated libraries:
xtool.exe inject _libraries\fast-lzma2.dll
xtool.exe inject _libraries\preflate_dll.dll
xtool.exe inject _libraries\zlibwapi.dll

And I want to replace only one a new version fast-lzma2.dll, then:
xtool.exe inject _libraries\fast-lzma2.dll

Or probably, every time there is a new version of xtool, you need to create a new integration.

3. Yes.

[Razor12911]

Originally Posted by Cesar82

@Razor12911, thank you very much for the constant updates.

1) Is it possible to update the previously injected libraries?
2) XTool 0.3.21 has been removed from the main post... Will we have updates for the remaining plugins soon?
3) Is it necessary to include the "fast-lzma2.dll" library to use xtool's lzma2 internal method?
4) What xtool method is the library "xdelta3_dll.dll" from the _libraries folder used for?

1) yes
2) possibly, in the main post there is actually link to the older releases I just removed 0.3.21 to make people not ask what is the different between this version and the recent update.
3) yes
4) imperfect streams use xdelta for patching, this function initially came from a dll but I separated all dlls from the main executable and gave the options to the user to include such a feature.

[elit]

I have found something that seem like a bug, and I am not sure if its FA or xtool, but likely an xtool issue.

Say you precomp a file with reflate, but you have an xtool.ini config that contain all kind of codecs, like xmemcompress, quickbms based and so on. But you only used reflate.

Now to decode successfully, you have to have all those exe files + untouched xtool.ini otherwise xtool will throw general error! Even if you don't need any of those for decompression.
That just happened to me. I had to copy all those pointless exe's and exact same xtool.ini for it to work. If for example I edited out those unneeded codecs from xtool.ini and/or deleted crilayla.exe or xmemcompress.exe, then xtool won't decode my reflate pack!

EDIT: No wonder it fails:
xtool.png
Why it have to save all those things if it's not needed?

[Razor12911]

Originally Posted by elit

I have found something that seem like a bug, and I am not sure if its FA or xtool, but likely an xtool issue.

Say you precomp a file with reflate, but you have an xtool.ini config that contain all kind of codecs, like xmemcompress, quickbms based and so on. But you only used reflate.

Now to decode successfully, you have to have all those exe files + untouched xtool.ini otherwise xtool will throw general error! Even if you don't need any of those for decompression.
That just happened to me. I had to copy all those pointless exe's and exact same xtool.ini for it to work. If for example I edited out those unneeded codecs from xtool.ini and/or deleted crilayla.exe or xmemcompress.exe, then xtool won't decode my reflate pack!

EDIT: No wonder it fails:
Attachment 32214
Why it have to save all those things if it's not needed?

Xtool stores all possible methods because xtool itself doesn't know if are or will be used if all before even precompression begins.

[elit]

Greetings Razor.
Back on page 34 where I wrote about reflate problems, causing crc errors... This time I was able to catch in on console:

reflatebug.png

But the thing is, exact same run may pass successfully on second or more tries(and then maybe fail again, so random dice per run). I am starting to be cautious about HW issue possibility on my side, but then I had no such issues with zlib yet. Using -t8.

Looks like either thread race issue, or my HW memory. This is on xtool v0.5.3. I will investigate this further.

EDIT:
Latest v0.6.2 == same issue.
If xtool do not fail during compression(precomp), then it will always recompress successfully during unpacking - i.e. probably not HW issue after all.
Last stable version without this bug that I have is 0.3.21. Cannot replicate on it. I believe this one is stable.[Its not, later I found..] Unfortunately it miss a lot of streams that newer xtool-reflate can see.
EDIT2:
Forgot to add. Changing chunk size(or not using depth) can help sometimes, but on big enough data stream one chunk size may pass at one point and fail at different one(where again some other size could work and so on). Also, I started having these issues only recently which correspond with me upgrading to later xtool version than I was stuck with.
Also, I think this have way more likelihood of happening on files that do not contain(or very little) deflate chunks. When FA reported CRC fails it was pretty much files that happen to share same extension with those that do.
EDIT3:
Yup, not my HW. I just threw whole Halo Master Collection directory to xtool 0.3.21(without filter) and it completed successfully. Then I tried both v0.5.3 and v0.6.2 and it failed on address violation.
EDIT4:
I will further confirm it later here, but using -lm *may* work as a workaround for reflate. Not due to a memory shortage, but the way chunks/data and threads are handled. No, it didn't help. [ADD: neither did -t1]
EDIT5:
Preflate works without issues, so I had to settle with it. Final repack size is ~1gb bigger though. I do hope reflate get fixed in the future as I prefer it.