XTool 2020 (Main Project)

[janb14]

Do you know any other ogg compressors ? Either way running it on single stream still resulted in a noticable reduction and i would see it as an example of how to correctly fill out these ini files. Can you help me with that ? From the xtool chm im not quite able to do that yet.

[Razor12911]

Update available

Changes

- Minor bug fixes

[infovs]

Originally Posted by Razor12911

Update available
Changes
- Minor bug fixes

Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?
And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only..
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards

[Razor12911]

Originally Posted by infovs

And thank you for including x86 version btw, I can use it on older x86 games to retain compatibility out of nostalgic reasons only..
Regarding this x86 compile, one addition small "info" question...in _x86\plugins you included x86 versions of hif2raw_dll.dll and raw2hif_dll.dll with datestamp(and compile time) 2013...tested it on Windows Vista x86 (as I said, no need to judge, I sometimes use this out of nostalgic and historic perseverance reasons) and they work...but you used to distribute 0.3.21 xtool with newer(are they?) x86 versions, 2017 compile time. They also work, I'm just confused, are those from 2013 included with x86 0.7.3 by mistake or is it better to use them instead of 0.3.21 x86 hif2raw_dll.dl/raw2hif_dll.dll versions?
Thanks. best regards

Initially only the x86 libraries existed and after the developer of reflate returned years after, he provided x64 versions however they function the same. (Also the libraries included in 0.3.21 are actually x64, that xtool release was also x64 )

Thanks Razor12911, just curious though, quite a big difference in exe size from previous version considering "minor bug fixes"...is this expected, different compiler or something?

I've actually added Denuvo to limit installation of xtool compressed games to five per day.

[infovs]

Originally Posted by Razor12911

I've actually added Denuvo to limit installation of xtool compressed games to five per day.

Glad to see you added older Denuvo version otherwise xtool.exe size would be like 1GB ..

Joking aside, regarding x86 hif2raw_dll.dll/raw2hif_dll.dll..you're right - 0.3.21 version did not contained x86, 0.3.19 did but only x86 xtool.exe.
x86 hif2raw_dll.dll/raw2hif_dll.dll libs versions I'm mentioning are contained with xtool_1908_R6, they seems newer than ones distributed with 0.7.3, hence the confusion

[kj911]

Originally Posted by Gehrman

Hi Razor12911
Please add -a option to Srep. Sometimes, for big games like FH5, -a0 should be used to use less memory.
...

Used -a0 in many times ago!

Try it -m5f mode, its minimally smaller memory eat than -m3f. And combined in -a0. This m5f:a0 smaller memory footprint from decompression than m3f:a0??

In last newer ~20kB sized hif/raw dlls, in have different results than older ~80kB sizd versions. 50% case bigger inflated/compressed results.
The one problematic parts, from x86 build packages, the "preflate_dll.dll" file, not compatible from WinXP and not working from any precompressors usage. (ex: ZREC) Still exist working one in have preflate library available in many precomp 0.4.8 and older versions. (x86 releases.)

[Razor12911]

Update available

Changes

- user can specify srep parameters to use via -sp#
- updated zlib codec

Notes

the -sm# parameter is removed, now if you want to control the memory srep is bound to use, you use refer to the list of srep's parameters and use it as such -spmem75p or whatever it is you prefer, -sp is also available when encoding in order to specify additional srep parameters like the user Gehrman has requested. -spa0:l512.

The executable got bigger because I've been busy trying to add some features to the program that have not come into fruition but if you're a nosy person, you can check the source code to see what caused the exe to get big.

[KaktoR]

What is wrong here? I don't get it.

Streams: 12050 / 12446
Time: 00:00:03 (CPU 00:00:18)

Size: 78.9 MB >> 211 MB

Compressed 2 files, 84,074,653 => 84,074,928 bytes. Ratio 100.00%
Compression time: cpu 0.06 sec/real 2.99 sec = 2%. Speed 28.16 mB/s
All OK

[External compressor:xtool_test]
header = 0
packcmd   = xtool.exe precomp -mscarletnexus+zlib -c128mb -t100p -d1 - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

[Gehrman]

Thank you Razor12911 for the new version


XTool 0.7.4 test
Forza Horizon 5

134 GB > 198 GB > 78.8 GB

Creating archive: FH5PE - 1.594.508 - 001.Bin using xtool:c32mb:mzlib:dd3:spa0:hash-+4x4:lzma:fb273:lc8:mfbt4:64m+diskspan:1395864372:1557135360
Compressed 17,216 files, 143,891,581,840 => 1,395,864,412 bytes. Ratio 0.97%
Compression time: cpu 477.42 sec/real 28649.30 sec = 2%. Speed 5.02 mB/s
All OK
CT: 07:57:29

Tested 17,216 files, 1,395,864,412 => 143,891,581,840 bytes. Ratio 0.97%   
Directory 171,831 => 873,633 bytes. Ratio 19.67%
Testing time: cpu 181.02 sec/real 6340.69 sec = 3%. Speed 22.69 mB/s
All OK

DT: 01:45:40 (In another compressed version that used XTool 2020, the extraction time was 1 hour. It seems that this version has decreased in decompression speed.)

a problem
Heavy CPU consumption in Normal CPU Priority mode, which causes Mouse Lag.
This problem does not exist in "Below Normal" CPU Priority.

[Razor12911]

The log information via xtool when repacking FH5:

Duplicates: 409792 (8.55 GB) [13.1 GB >> 22.2 GB]
Srep decompression memory: 5.30 GB [19.9 GB*]

Something you have to keep in mind when repacking Forza Horizon 5 is that this game has a lot of duplicates because its also a big game. If you use srep, it will request 20 GB for decompression memory or (5 GB if you used -dd while xtool itself will request 8.5 GB, about 13 GB overall, ~7 GB memory savings) and since you made a post that you wanted to control srep's a# parameter, I can only assume that when you ran decompression before, you had more system memory available than when you ran it this time.

The speed reductions seems to be related to you not having enough memory to install the game without srep or xtool making virtual temp files which incur speed penalty.

Games like these are the reason I added GPU memory feature in xtool because it would offload some memory onto the GPU which typically isn't used when installing games (if you have 8GB GPU, xtool would require 1GB instead of 8.5 GB, while 7.5GB will be on the GPU then installation would need 6GB instead of 20 GB which srep initially required), though not ready because it's not thoroughly tested on a lot of systems to see if it works without issues.

I ran a test between 0.7.4 against older releases, they all produce the same results and have the same speeds when decoding (at least within margin of error) and 0.7.0+ encodes faster. Decoding speed in 0.7.4 is slightly slower but reasonable because of memory usage optimisations that were made in 0.7.2, all versions used 220 MB+ for decoding while 0.7.4 used 73 MB.

ForzaHorizon5\DLC\MicrosoftExpansion2FH5\media\Tra cks\Falcon\bin.zip

0.7.4
Compressed 1 file, 1,893,356,830 => 2,847,735,336 bytes. Ratio 150.41%
Compression time: cpu 1.36 sec/real 20.95 sec = 6%. Speed 90.39 mB/s

Tested 1 file, 2,847,735,336 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.19 sec/real 22.40 sec = 1%. Speed 84.52 mB/s

0.7.0
Compressed 1 file, 1,893,356,830 => 2,847,735,336 bytes. Ratio 150.41%
Compression time: cpu 1.41 sec/real 20.57 sec = 7%. Speed 92.05 mB/s

Tested 1 file, 2,847,735,336 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.33 sec/real 22.06 sec = 1%. Speed 85.84 mB/s

0.6.9
Compressed 1 file, 1,893,356,830 => 2,847,735,092 bytes. Ratio 150.41%
Compression time: cpu 1.48 sec/real 25.08 sec = 6%. Speed 75.50 mB/s

Tested 1 file, 2,847,735,092 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.14 sec/real 21.56 sec = 1%. Speed 87.81 mB/s

0.6.5
Compressed 1 file, 1,893,356,830 => 2,847,735,092 bytes. Ratio 150.41%
Compression time: cpu 1.38 sec/real 24.58 sec = 6%. Speed 77.03 mB/s

Tested 1 file, 2,847,735,092 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.30 sec/real 22.06 sec = 1%. Speed 85.83 mB/s

0.5.3
Compressed 1 file, 1,893,356,830 => 2,847,735,104 bytes. Ratio 150.41%
Compression time: cpu 1.39 sec/real 25.08 sec = 6%. Speed 75.51 mB/s

Tested 1 file, 2,847,735,104 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.16 sec/real 21.56 sec = 1%. Speed 87.81 mB/s

0.4.0
Compressed 1 file, 1,893,356,830 => 2,847,735,104 bytes. Ratio 150.41%
Compression time: cpu 1.38 sec/real 24.39 sec = 6%. Speed 77.64 mB/s

Tested 1 file, 2,847,735,104 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.41 sec/real 21.56 sec = 2%. Speed 87.82 mB/s

[Cesar82]

@Razor12911, can the new -sm# parameter be used from the freearc command line using comma "," to separate srep parametters?

xtool:mzlib:dd3:spa0,s100gb

I would also like to know if running xtool with the -g90p parameter on a system without a dedicated video card will also work or could it crash the system?

Thanks!

[Razor12911]

Update available

Changes

- added footer feature in configuration plugins
- added thread priority controller via -T# parameter

Notes

Footer detection has been added to configuration which means if there's no way of knowing the size of a stream but if within the stream itself has a footer or end of file/stream marker, you can input that as a signature for xtool to look for which will help you in determining the size of the stream.

For example, jpeg image file has the header 0xE0FFD8FF but if you looked at the last 3 bytes, you'll notice that in all of them, they end with 99 FF D9, this will be the footer and your configuration should look something like this

[Stream1]
Name=jpg
Codec=packjpg
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

The offset is -4 because the signature is part of the stream, you have Footer=0xD9FF99 which xtool will search for after finding Signature=0xE0FFD8FF, the compressed size is then Footer + 7 because the signature + the footer are both part of the stream.

Another use case for this is if you have streams stored in sequence, if their signature is 59 61 7A 30 and the next stream has the same signature, you can write the next stream as the "footer" of the current stream.

[Stream1]
Name=Yaz0
Codec=Yaz0
BigEndian=0
Signature=0x307A6159
Footer=0x307A6159
Structure=Signature(4),Stream,Footer(4)
StreamOffset=-4
CompressedSize=Footer+4
DecompressedSize=0

This time, the compressed size doesn't added both signature and footer because the footer is actually the header of the next stream, there's one caveat with doing it this way however, the last stream will not be captured.

Thread priority option is added and can be used via -T and the values are 0..6 with zero being idle and 6 being real-time (time critical), if you have laggy mouse while installing a game knowing full well that you have set 100p for xtool and have lolz or lzma working and toasting your CPU and saying it's xtool's problem then you should use this but know that setting this too high means your system will be unresponsive or setting too low making installation longer, well... up to you. Too many notes...

@Cesar82

You can use commas to separate parameters.

For GPU, xtool performs a check to see if the system has a GPU with at least 1GB VRAM and even then it makes sure that there's at least VRAM 512MB left unused. If the condition is not met, it doesn't use GPU at all.

[Razor12911]

Update available

Changes

- Minor bug fixes

Notes

This release contains recompiled zlibwapi.dll, preflate_dll.dll, lzo2.dll (to remove Visual C++ 2010 requirement) and xdelta3_dll.dll simply because there are people who are using Windows XP in this day and age, shocking I know...

[shazzla]

Damn. I'm lame

How can i use this with XTool076 for JPG stream compression?

[Stream1]
Name=jpg
Codec=brunsli
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

Solved :
My sample JPG have different header.

[infovs]

Originally Posted by Razor12911

Update available
Changes
- Minor bug fixes
Notes
This release contains recompiled zlibwapi.dll, preflate_dll.dll, lzo2.dll (to remove Visual C++ 2010 requirement) and xdelta3_dll.dll simply because there are people who are using Windows XP in this day and age, shocking I know...

Thank you Razor12911 for your updated xtool and libraries...anyway, tested unpacking speed of your last three x64 zlibwapi.dll library compiles, multiple times (did not test x86 version at all)...

Unpacking speed of zlibwapi.dll 1.2.11/2020-07-18/535552 bytes is _always_ faster than any of your other, newer, 1.2.13 compiles (2022-12-31/544768 bytes and 2023-07-11/512000 bytes)
Newest XP compatible zlibwapi.dll is by far slowest during unpacking, probably because it does not use some of advanced CPU instructions available after XP. Anyway, I truly welcome XP support because I consider games that are playable on XP should be able to unpack on XP...but if making .dlls XP compatible comes with obligatory speed loss than maybe it is better to provide separate XP compiled libs, just my thoughts of course. Ideally, one compile should work on all OSes with max speeds but don't really sure if this is possible...

Following are examples of decompression speed of _same_ arc packed with xtool:zlib+srep:m3f+lolz ,all that is changed is zlibwapi.dll before unpacking (encoding speed is not primary important to me, arc is packed using one of mentioned zlibwapi.dll but this does not impact decompression speed, tested)
Tested on Windows 11, 20 threads CPU, xtool 0.7.4 used, -t100p in arc.ini

Using zlibwapi.dll 1.2.11/2020-07-18/535552 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 0.97 sec/real 74.49 sec = 1%. Speed 62.87 mB/s
All OK

Using zlibwapi.dll 1.2.13/2022-12-31/544768 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 1.59 sec/real 83.01 sec = 2%. Speed 56.42 mB/s
All OK

Using zlibwapi.dll 1.2.13/2023-07-11/512000 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 0.63 sec/real 106.93 sec = 1%. Speed 43.80 mB/s
All OK

Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it..

Regards