XTool 2020 (Main Project)

[shazzla]

Doc (.chm) available in the xtool package.

[Razor12911]

Update available

Changes

- added advanced configuration based plugin support
- added UI mode when xtool.exe is launched with xtoolui.dll present
- added skip verification mode
- xtool now enforces w15 deflate stream detection by default
- fixed oodle scanner exceptions when incomplete stream is detected
- fixed issue with zlib codec not accepting streams from database plugins
- updated command line syntax

Notes

advanced configuration plugin is a way of writing more complex ini files for stream detection without the need of coding skills or the need of creating a library based plugin.

spidey plugin is an example, regular configuration files could not add support for this game however, advanced configuration allows this and here's an example of how that looks like

[StreamList1]
Name=lz4
Codec=lz4
BigEndian=0
Signature=0x1000352415344
Structure1=Signature(8),Count(4),Unk1(4),Unk2(16)
StructureN=DPos(8),CPos(8),DSize(4),CSize(4),Unk3(8)
StructureS=Stream
CounterStart1=1
CounterEnd1=Count
CounterStep1=1
StreamPosition=CPos
StreamOffset=0
CompressedSize=CSize
DecompressedSize=DSize
Condition1=

The user interface mode has been added and it's not meant to compete with tools like DiskSpanGUI or other similar programs but it's meant to help newbies operate the program as everything is made as simple as possible. The program can still be useful for regular users too, you can use the user interface mode to get theoretical outputs without doing actual (pre)compression, if you set output to none (leave blank in cli mode) and enable skip verification then essentially, you are performing a scan on the input (similar to what Drop and Scan for Zlib does) but for all supported codecs.

An example is Cyberpunk 2077 as shown below.
xt1.PNG
xt2.PNG

From here, you can find out the output size if you were to do precompression (in about 5 mins or less, all depends on drive speed).

More uses? Well people have a problem deciding how much chunk size to use, you can change the chunk size and see how the stream found or how the size differs to decide what chunk size to use.

The UI is a bit borked at the moment but the next update will improve upon it.

xtool enforces w15 streams because I figured out that this is what causes reflate to produce crc errors because there are more false positives this way. So how does this affect results, it doesn't because 99% of games use w15 anyways except Dishonored 2 and Dishonored Death of the Outsiders (set w10 for them)

Syntax has been updated as per request from Cesar82:
--dedup can also be -dd
--dedup=#, -dd#
--dbase, -db
--diff=#,-df#
--verbose, -v
--skip, -s

[Razor12911]

Update available

Changes

- added library checker (trial and error)
- improved user interface
- fixed bugs related to oodle scanner
- skip verification no longer applies to encryption codecs

Notes

Library checker allows you to find out what library was used through trial and error, all you do is pick a directory with a list of either lz4, zstd or oodle libraries and set output to none then every single library found in that directory is loaded by xtool one by one while showing you how many streams were processed by each and their respective precompressed outputs. This should allow you to maximize compression in one click rather than doing it manually.
xt3.PNG
xt4.PNG

More syntax changes as per request from Cesar82:
--zlib= can also be -zb
--lz4=#, -l4#
--lzo=#, -lo#
--zstd=#, -zs#
--oodle=#, -od#
--srepmem=75p (when decoding) as requested by Gehrman

Announcement

This is the last update for xtool, I am at a point where I think I have done enough for this project. I have dragged development of the project longer than I should have but I guess it's a habit of mine of not leaving something unfinished and this is the creative vision I had for this tool from the start, it took longer than I expected but I'm glad it's done.

So what does that mean I am leaving the forum? No, I'll stick around for the time being, it's just that the main project is no longer getting updates... and I had to stop at the magic number 69 . So if there are bugs and issues, you'd have to refer to the older releases which are made available on the main post.

[dixen]

Xtool v0.6.9 Elden Ring Edition

Two xtool.exe with injected oo2core_5_win64.dll and oo2core_8_win64.dll

Compressing 1 file, 2,655,581,216 bytes
Compressing Data3.bdt
Compressed 1 file, 2,655,581,216 => 4,846,074,313 bytes. Ratio 182.49%
Compression time: cpu 4.28 sec/real 347.49 sec = 1%. Speed 7.64 mB/s
All OK

[External compressor:xt8,xt5]
header = 0
packcmd = {compressor} precomp -mkraken:l6 -c32mb -t100p - - <stdin> <stdout>
unpackcmd = {compressor} decode -t100p - - <stdin> <stdout>

All with Happy New Year!!! Be Happy!!

[Hexagon123]

Tried using LZ4, but no streams.
Anyone make a plugin for this XTool.

Seen Puyo Puyo Tetris 2, Sonic Frontiers, and a few titles that uses this compression.

Ref1:http://wiki.xentax.com/index.php/Sonic_Games_PAC
Ref2:https://github.com/nickworonekin/puyo-pac

[Razor12911]

Originally Posted by Hexagon123

Tried using LZ4, but no streams.
Anyone make a plugin for this XTool.

Seen Puyo Puyo Tetris 2, Sonic Frontiers, and a few titles that uses this compression.

Ref1:http://wiki.xentax.com/index.php/Sonic_Games_PAC
Ref2:https://github.com/nickworonekin/puyo-pac

The sample you've provided is too small and only contains one stream, can't make a plugin off of this.

XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 000000000000081F (2080)
[0] Actual lz4 stream found at 0000000000000040 (2005 >> 9880)

[0] Processing streams on block 0000000000000000 to 000000000000081F (2080)
[0] Processing lz4 stream at 0000000000000040 (2005 >> 9880 >> 2171) using a0 has failed
[0] - Patching stream at 0000000000000040 (2005 >> 2171) [1151] has failed

Streams: 0/1
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)

[elit]

"I had to stop at the magic number 69.."
^So you are a Scorpio

Anyway I am glad you made my lz_scan (almost)obsolete because this is way better. I will later check in more detail how to do a multi-lib scan. From the pictures it seems you create a version directory and put a dll there, like 2.8.1/codec.dll, 2.8.2/codec.dll and so on, while dlls are all of same name.
As opposite to my lz_scan where I had a single directory with like liblz4_112.dll, liblz4_120.dll etc.

Advanced config is what I love about this new version and will use, really kudos for that.

I recommend if you want to call it a day, don't add a new features anymore but hang around to collect and fix last remaining bugs that users report over time. What you achieved here is a project that gives any institutional budget projects run for their money. I say lets not leave it half backed, after all you do remember Freearc(and srep) fiasco. Both a buggy, half assed projects, but no proper alternative unfortunately(at least for my usage as a GUI archiver).

After some time once project is bug-hardened with no more changes, maybe I try to port it one day to C and/or multiplatform if I decide to switch to Linux in the future. Thank you for all you did for us so far.

[Wanterlude]

Good afternoon, everyone!
I decided to pack Transport Fever 2 using zlib+dedup+lolz, but srep takes up all RAM when unpacking. I looked on the forum to see if this was a problem before and saw the --mem=# parameter. I tried "20p" "800m" "800mb" but nothing ended up working. This is what I have prescribed in arc.

[External compressor:xtool]
header    = 0
default   = -c128mb -t100p
packcmd   = xtool.exe precomp { -moption} -db -dd5 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p --mem=10p - - <stdin> <stdout>

xtool - last ver.
srep 32-bit v392
ASIS script (i try v732,v744)

[Razor12911]

it should be --srepmem=...

[elit]

I would like to bring to your attention an issue I am currently experiencing with xtool.

In the past, I compressed several games with it under Windows 7. Probably most used version by me back then was 047 and 053. I test every archive after creation and back then every one of them tested fine.

However about 2 months ago I switched to Windows 8.1 and I find myself facing decompression errors.
Specifically this almost exclusively relate to reflate(I had one zlib giving me error as well but not all). It is 100% reproducible and do not matter which xtool version I use(I keep ones I compressed anything with just in case). Reflate in archive = 100% fail. Any archive. Due to that I excluded disk issue.

Note that ztool with -cm0 work fine under Win8.1, as well as 32bit xtool-crilayla version. Problem is only xtool->reflate(maybe occasional zlib) from what I seen so far. Thankfully majority of my archives are ztool based due to me stubbornly sticking to it for so long but still quite few of new ones I made use xtool already. xtool doesn't have any fancy switches like -cm0, I tried -t1 to no avail.

I am currently trying all archives to make list of broken ones, I also already installed Win7 under Hyper-V late night and will try more on it later but don't know if environment will be 1:1, maybe xtool is having issues due to memory/exception handling of Win8.1.
Also, I did of course already tried Win7 compatibility(all relevant exe files in chain) as well as running as an admin(real one, enforced through group policy).

If anyone have an idea, I will appreciate it.

[elit]

Originally Posted by Gehrman

Use compatibility mode
Setup.bat

ECHO OFF
Reg.exe add "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe" /t REG_SZ /d "~ DPIUNAWARE WIN7RTM" /f
START Setup.exe
Reg Delete "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe"  /f
Exit

That's interesting, so it seems a known issue. Thank you, I will try later after I scan all my archives and report back. I hope this will really fix it.

[elit]

^Unfortunately, putting both freearc.exe and xtool.exe into registry like above did not help.err.png

EDIT: Not all reflate are broken, in fact many are fine. I am starting to see potential pattern but will confirm it later whether its case. Ones with the issue so far are the ones with custom settings, like chunk size -c32m etc., whereas all the good ones are just default -mxtool:reflate so far.

EDIT2: I see the pattern. This error affect both reflate and zlib, when high chunk size(>32mb) is used. This was working fine on Win7. Changing thread number doesn't seem to cut it. And I have same rig with enough memory(16G). I will try few more things but it seems as if Win8.1 is handling memory or relevant functions differently.

EDIT3: Windows 7 under Hyper-V did not help. I still believe archives are good and this is because I don't run under native.

[Gehrman]

@elit
Increase the size of the Windows page file (pagefile.sys).

[Razor12911]

could you try isolating the bug and send over the sample that causes xtool to bug out

[elit]

@Gehrman
I have 16G RAM + 16G swap. I also tried -t1 option. This is certainly not the case of RAM but also not of disk space.

@Razor12911
I have looked into it today and it does look like it may have to do with chunk size. All archives that bugged out had -c more than 32mb(64m, 128m), but those without any option or up to 32m were all fine. Best at this point will be for you to see yourself. I will upload 2 of them + FA whole pack that I use. With that you will be able to understand how I use it and try for yourself. Again this all worked under W7 native install. Also I am sure among those hundreds of archives where ztool was used, there had to be at least several with chunk size >32m and those passed fine(I tested my whole collection).
I will upload things somewhere and post the links by tomorrow.