XTool 2020 (Main Project)

[Masquerade]

Originally Posted by yasitha

what is this ?

https://fileforums.com/showpost.php?...&postcount=493

[elit]

During repacking of Final Fantasy XIII(PC version), I found that using

xtool:d1:c64m:mzlib:mreflate

cause crc error during unpacking of the game. Cause is almost certainly due to combining zlib with reflate as alone either one work. Maybe they are not together compatible with depth option, only d0?

[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t50p {options} - - <stdin> <stdout> 
unpackcmd = _EC\xtool\xtool decode -t50p          - - <stdin> <stdout>

EDIT: Its probably something else, I still got error today after using only reflate. Its strange I don't tend to get these kind of errors for no reason:

fferr.png
fferr2.png


-----------------------------------------------------------------------------------
EDIT2:
Found the problem! It's with reflate. If I use zlib instead:

xtool:d1:c32m:mzlib+srep64:m3f:mem8g:a0+4x4:lzma:64mb:normal:32:lc8

, it works. I don't know why, in other cases I never saw issue with reflate.

ADDENUM: It's due to depth. Anything higher than -d0 cause failure during decompression. Zlib and preflate do not have this problem. Data tested is 'sys' directory(2.8gb) in Final Fantasy XIII(PC, fitgirl version).

[Cesar82]

Originally Posted by elit

During repacking of Final Fantasy XIII(PC version), I found that using

xtool:d1:c64m:mzlib:mreflate

cause crc error during unpacking of the game. Cause is almost certainly due to combining zlib with reflate as alone either one work. Maybe they are not together compatible with depth option, only d0?

[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t50p {options} - - <stdin> <stdout> 
unpackcmd = _EC\xtool\xtool decode -t50p          - - <stdin> <stdout>

I could be wrong, but using this method line, I think something like this should be used in arc.ini:

[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t50p { -option} --dbase - - <stdin> <stdout>
unpackcmd = _EC\xtool\xtool decode -t50p - - <stdin> <stdout>

[elit]

Originally Posted by Cesar82

I could be wrong, but using this method line, I think something like this should be used in arc.ini:

Nope, still throws the error. Also it compress fine and show correct chain on archive info:

xtool:d1:c64mb:mzlib:mreflate+srep64:m3f:mem8g:a0+4x4:lzma:64mb:normal:32:lc8

, which is same regardless if I use {options} or { -option}

EDIT: Found the problem, see 2 posts above.

[elit]

Also, aside from what I stated above, there was another interesting thing I found. When inflating(with reflate option), if I use c32m-c128m range, xtool process about 25150 streams. Then when I use -c256m, it only process ~22000 streams. And yet, if I use -c512m option, it again process ~25150 streams. All other options same. Go figure.

[elit]

And even more interesting, reflate is actually faster than zlib at same options(with depth 1 and chunk size of 32m at least)???:

zlib.png
reflate.png

[elit]

And preflate is worse than zlib hmmm..:

ff13pref.png

[Razor12911]

Update available

Changes

- added wav stream detector
- added flac codec
- added jpg stream detector
- added packjpg, brunsli, jojpeg codec
- added feature that allows input to be a directory
- added feature to extract detected streams
- updated database feature
- updated deduplication feature
- IO function decode updated

Notes

I have added wav audio compression as an alternative of msc for people who has issues with it or don't want to use Freearc's tta codec as it processes wav files individually. One thing to note, it's not as good as tta, tak or even frog as it is based on flac codec (It was the only open source codec I could work with...).

Also added jpg image compression codecs, you can pick between packjpg, brunsli or jojpeg. packjpg seems to have a memory leak, brunsli is fast but cannot deal with large jpg images and jojpeg can be used if you're after ratio as it is paq based (seems to be buggy so stick to packjpg or brunsli for now)

A new parameter is added --extract=[path], this will extract all detected streams to a directory... if you're interested in the streams themselves.

database and deduplication features have been updated and can now be used at all times without worrying about crc collisions.

Special thanks to KaktoR for running several tests and Shelwien for providing compiled libraries for brunsli and jojpeg.

@elit
These issues will be investigated in time.

[dixen]

Find little uncritical bug

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 1/1
Time: 00:00:02 (00:00:02)
Memory: 276 MB (276 MB)

100%
Errorlevel=0
Compressed 1 file, 77,790,338 => 53,378,764 bytes. Ratio 68.62%
Compression time: cpu 0.08 sec/real 2.78 sec = 3%. Speed 28.01 mB/s
All OK

Compressing 77,790,338 bytes with xtool.exe precomp -mFLAC --verbose -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 0000000001FFFFFF (33554432)
EAccessViolation: Access violation at address 0000000000409A77 in module 'xtool.exe'. Read of address 00007FF4FE6BFBA4

Errorlevel=1

ERROR: general (de)compression error in xtool

Anyway, thank Razor for update and new functions)

One question. Why TTA but not TAK?

Original.wav 74.1 mb
ENCTTA.tta 50.3 mb
ENCTAK.tak 48.7 mb

[Masquerade]

Originally Posted by dixen

One question. Why TTA but not TAK?

TAK isn't Open Source.

[Razor12911]

Originally Posted by dixen

Find little uncritical bug

your chunk size is too small for that input

Originally Posted by Prince4

Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

0.3.21 is an old version that supported old plugins and it is kept there until all plugins are updated to support 0.4+ versions, use 0.6.0 if you are not using old plugins.

Originally Posted by elit

Interesting, I should note that compression difference between different wav codecs is minimal.

https://wiki.hydrogenaud.io/index.ph...ess_comparison

I picked codecs based on the information provided here, it's not that I haven't tried wavpack. I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

Originally Posted by elit

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

That's pretty much how it is intended to work, wav files are scanned within files and then fed to codecs.

[elit]

Originally Posted by Razor12911

I did but it requires additional code as it generates lossy wav and a patch file in case you wanted to restore the original pcm data whereas flac does not. In addition, it doesn't seem to be thread safe which means it cannot work in xtool environment (maybe I'm mistaking it with tta).

That is optional feature with extra switches, by default it work same as flac - no lossy wav. As for thread safety, I believe its single threaded app so could be used chunk per thread but I don't know for sure.
I am also perfectly ok with flac as well so don't worry about it, I just though this one may have been better due to more formats supported.
https://www.wavpack.com

[elit]

Interesting, I should note that compression difference between different wav codecs is minimal. Also for me in the past, I found wavpack the best, It handle more formats(WAV/BWF/RF64, Sony Wave64, Apple CAF, Philips DSDIFF, and Sony DSF (all with no size limitations)), more modes(8, 16, 24, and 32-bit integer PCM; 32-bit float PCM; DSD audio; mono, stereo, and multichannel; sampling rates from 6 to 192 kHz (and non-standard rates)), RIFF chunks, is open source, does support STDIO which for me work unlike flac, which for some reason during decompression kept running past original size inflating to infinity(when used in FA with stdio mode).

However, flac is great too because is non-symetric so decompression is fast(although wavpack also seem to support asym. cmp.).

I wonder how this work, if xtool is able to use it on chunks when wav files are inside big game packs, that would be indeed very useful. In fact, I don't know what would be the point beyond that as we could then just use standalone codecs?

[Prince4]

Could someone please remind me what's the difference between the 0.3.21 and 0.6.0 versions?

[Razor12911]

Update available

Changes

- added fast lzma2 compression for portable mode
- fixed issues with wav stream detection
- fixed minor issue with stream deduplication feature

Notes

I have added fast lzma2 compression for users who would want to use xtool without FA but still want to perform compression immediately after precompressing.

Example

xtool.exe precomp -mzlib -c32mb -t100p --dbase --dedup --compress=l10,t100p - -