XTool 2020 (Main Project)

[Razor12911]

The log information via xtool when repacking FH5:

Duplicates: 409792 (8.55 GB) [13.1 GB >> 22.2 GB]
Srep decompression memory: 5.30 GB [19.9 GB*]

Something you have to keep in mind when repacking Forza Horizon 5 is that this game has a lot of duplicates because its also a big game. If you use srep, it will request 20 GB for decompression memory or (5 GB if you used -dd while xtool itself will request 8.5 GB, about 13 GB overall, ~7 GB memory savings) and since you made a post that you wanted to control srep's a# parameter, I can only assume that when you ran decompression before, you had more system memory available than when you ran it this time.

The speed reductions seems to be related to you not having enough memory to install the game without srep or xtool making virtual temp files which incur speed penalty.

Games like these are the reason I added GPU memory feature in xtool because it would offload some memory onto the GPU which typically isn't used when installing games (if you have 8GB GPU, xtool would require 1GB instead of 8.5 GB, while 7.5GB will be on the GPU then installation would need 6GB instead of 20 GB which srep initially required), though not ready because it's not thoroughly tested on a lot of systems to see if it works without issues.

I ran a test between 0.7.4 against older releases, they all produce the same results and have the same speeds when decoding (at least within margin of error) and 0.7.0+ encodes faster. Decoding speed in 0.7.4 is slightly slower but reasonable because of memory usage optimisations that were made in 0.7.2, all versions used 220 MB+ for decoding while 0.7.4 used 73 MB.

ForzaHorizon5\DLC\MicrosoftExpansion2FH5\media\Tra cks\Falcon\bin.zip

0.7.4
Compressed 1 file, 1,893,356,830 => 2,847,735,336 bytes. Ratio 150.41%
Compression time: cpu 1.36 sec/real 20.95 sec = 6%. Speed 90.39 mB/s

Tested 1 file, 2,847,735,336 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.19 sec/real 22.40 sec = 1%. Speed 84.52 mB/s

0.7.0
Compressed 1 file, 1,893,356,830 => 2,847,735,336 bytes. Ratio 150.41%
Compression time: cpu 1.41 sec/real 20.57 sec = 7%. Speed 92.05 mB/s

Tested 1 file, 2,847,735,336 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.33 sec/real 22.06 sec = 1%. Speed 85.84 mB/s

0.6.9
Compressed 1 file, 1,893,356,830 => 2,847,735,092 bytes. Ratio 150.41%
Compression time: cpu 1.48 sec/real 25.08 sec = 6%. Speed 75.50 mB/s

Tested 1 file, 2,847,735,092 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.14 sec/real 21.56 sec = 1%. Speed 87.81 mB/s

0.6.5
Compressed 1 file, 1,893,356,830 => 2,847,735,092 bytes. Ratio 150.41%
Compression time: cpu 1.38 sec/real 24.58 sec = 6%. Speed 77.03 mB/s

Tested 1 file, 2,847,735,092 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.30 sec/real 22.06 sec = 1%. Speed 85.83 mB/s

0.5.3
Compressed 1 file, 1,893,356,830 => 2,847,735,104 bytes. Ratio 150.41%
Compression time: cpu 1.39 sec/real 25.08 sec = 6%. Speed 75.51 mB/s

Tested 1 file, 2,847,735,104 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.16 sec/real 21.56 sec = 1%. Speed 87.81 mB/s

0.4.0
Compressed 1 file, 1,893,356,830 => 2,847,735,104 bytes. Ratio 150.41%
Compression time: cpu 1.38 sec/real 24.39 sec = 6%. Speed 77.64 mB/s

Tested 1 file, 2,847,735,104 => 1,893,356,830 bytes. Ratio 150.41%
Testing time: cpu 0.41 sec/real 21.56 sec = 2%. Speed 87.82 mB/s

[Cesar82]

@Razor12911, can the new -sm# parameter be used from the freearc command line using comma "," to separate srep parametters?

xtool:mzlib:dd3:spa0,s100gb

I would also like to know if running xtool with the -g90p parameter on a system without a dedicated video card will also work or could it crash the system?

Thanks!

[Razor12911]

Update available

Changes

- added footer feature in configuration plugins
- added thread priority controller via -T# parameter

Notes

Footer detection has been added to configuration which means if there's no way of knowing the size of a stream but if within the stream itself has a footer or end of file/stream marker, you can input that as a signature for xtool to look for which will help you in determining the size of the stream.

For example, jpeg image file has the header 0xE0FFD8FF but if you looked at the last 3 bytes, you'll notice that in all of them, they end with 99 FF D9, this will be the footer and your configuration should look something like this

[Stream1]
Name=jpg
Codec=packjpg
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

The offset is -4 because the signature is part of the stream, you have Footer=0xD9FF99 which xtool will search for after finding Signature=0xE0FFD8FF, the compressed size is then Footer + 7 because the signature + the footer are both part of the stream.

Another use case for this is if you have streams stored in sequence, if their signature is 59 61 7A 30 and the next stream has the same signature, you can write the next stream as the "footer" of the current stream.

[Stream1]
Name=Yaz0
Codec=Yaz0
BigEndian=0
Signature=0x307A6159
Footer=0x307A6159
Structure=Signature(4),Stream,Footer(4)
StreamOffset=-4
CompressedSize=Footer+4
DecompressedSize=0

This time, the compressed size doesn't added both signature and footer because the footer is actually the header of the next stream, there's one caveat with doing it this way however, the last stream will not be captured.

Thread priority option is added and can be used via -T and the values are 0..6 with zero being idle and 6 being real-time (time critical), if you have laggy mouse while installing a game knowing full well that you have set 100p for xtool and have lolz or lzma working and toasting your CPU and saying it's xtool's problem then you should use this but know that setting this too high means your system will be unresponsive or setting too low making installation longer, well... up to you. Too many notes...

@Cesar82

You can use commas to separate parameters.

For GPU, xtool performs a check to see if the system has a GPU with at least 1GB VRAM and even then it makes sure that there's at least VRAM 512MB left unused. If the condition is not met, it doesn't use GPU at all.

[Razor12911]

Update available

Changes

- Minor bug fixes

Notes

This release contains recompiled zlibwapi.dll, preflate_dll.dll, lzo2.dll (to remove Visual C++ 2010 requirement) and xdelta3_dll.dll simply because there are people who are using Windows XP in this day and age, shocking I know...

[infovs]

Originally Posted by Razor12911

Update available
Changes
- Minor bug fixes
Notes
This release contains recompiled zlibwapi.dll, preflate_dll.dll, lzo2.dll (to remove Visual C++ 2010 requirement) and xdelta3_dll.dll simply because there are people who are using Windows XP in this day and age, shocking I know...

Thank you Razor12911 for your updated xtool and libraries...anyway, tested unpacking speed of your last three x64 zlibwapi.dll library compiles, multiple times (did not test x86 version at all)...

Unpacking speed of zlibwapi.dll 1.2.11/2020-07-18/535552 bytes is _always_ faster than any of your other, newer, 1.2.13 compiles (2022-12-31/544768 bytes and 2023-07-11/512000 bytes)
Newest XP compatible zlibwapi.dll is by far slowest during unpacking, probably because it does not use some of advanced CPU instructions available after XP. Anyway, I truly welcome XP support because I consider games that are playable on XP should be able to unpack on XP...but if making .dlls XP compatible comes with obligatory speed loss than maybe it is better to provide separate XP compiled libs, just my thoughts of course. Ideally, one compile should work on all OSes with max speeds but don't really sure if this is possible...

Following are examples of decompression speed of _same_ arc packed with xtool:zlib+srep:m3f+lolz ,all that is changed is zlibwapi.dll before unpacking (encoding speed is not primary important to me, arc is packed using one of mentioned zlibwapi.dll but this does not impact decompression speed, tested)
Tested on Windows 11, 20 threads CPU, xtool 0.7.4 used, -t100p in arc.ini

Using zlibwapi.dll 1.2.11/2020-07-18/535552 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 0.97 sec/real 74.49 sec = 1%. Speed 62.87 mB/s
All OK

Using zlibwapi.dll 1.2.13/2022-12-31/544768 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 1.59 sec/real 83.01 sec = 2%. Speed 56.42 mB/s
All OK

Using zlibwapi.dll 1.2.13/2023-07-11/512000 bytes:
Extracted 4 files, 2,649,948,418 => 4,683,666,810 bytes. Ratio 56.58%
Extraction time: cpu 0.63 sec/real 106.93 sec = 1%. Speed 43.80 mB/s
All OK

Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it..

Regards

[shazzla]

Damn. I'm lame

How can i use this with XTool076 for JPG stream compression?

[Stream1]
Name=jpg
Codec=brunsli
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

Solved :
My sample JPG have different header.

[kj911]

infovs: Please testing more version older dlls...

Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.

This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.

[infovs]

Originally Posted by kj911

infovs: Please testing more version older dlls...
Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.
This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.

I tested all zlibwapi.dll versions that I could find since ztool..Razor12911 version from 2020-07-18 (535552 bytes) was the fastest at decompression. I also did my own compile of zlibwapi.dll 1.2.11 with MS VC 2010 and also 1.2.13 with MS VC 2010/2015/2022 -> VC 2022 1.2.13 compile produced best result but Razor12911 version was still fastest at unpacking, although not by much, speeds were pretty close. Actually only this latest 2023-07-11/512000 bytes version speed drop was noticeable for me, that's why I posted my observation.
Also, if you use MS VC and want your compile to be XP compatible you are pretty much forced to use VC 2010..yes, you can use XP platform toolset on some later MS VC versions but I had some issues with it. Never compiled with MinGW though so I can't comment on that.

[Razor12911]

vc22_vista_1.2.13: 24688ms 24687ms 24750ms 24750ms 24782ms
vc17_winxp_1.2.13: 24375ms 24172ms 24156ms 24266ms 24359ms
rad_studio_1.2.13: 24359ms 24219ms 24203ms 24265ms 24313ms

vc19_vista_1.2.11: 21734ms 21672ms 21610ms 21843ms 21578ms
vc22_vista_1.2.11: 21391ms 21359ms 21391ms 21343ms 21282ms
vc17_winxp_1.2.11: 21453ms 21469ms 21359ms 21328ms 21360ms

vc22_vista_1.2.8: 21282ms 21281ms 21266ms 21281ms 21281ms

zlib-ng_1.2.13: 7843ms 7797ms 7781ms 7813ms 7828ms

Originally Posted by infovs

Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it

I didn't do anything to 1.2.11 to make it fast, 1.2.13 itself is the one that is slow as you can see with the benchmarks, winxp compile has little to no impact on performance.

There's an issue opened on github regards to the speed which can be found here.
https://github.com/madler/zlib/issues/820

There was also a security vulnerability discovered in 1.2.11 and I don't know if patching this issue is what made zlib slower or not moving forward.
https://www.cybersecurity-help.cz/vdb/gnu/zlib/1.2.11/

The beauty of xtool is that the user can swap libraries if they prefer to use their own, there's also zlib-ng (utilizes modern CPU features) benchmarks added, as you can see it is faster however I doubt it produces matching crc as standard zlib but good to know nonetheless.

[Razor12911]

Update available

Changes

- added universal scanner for DirectStorage gdeflate streams
- added the use of gpu for caching and virtual memory purposes
- updated depthing feature to improve stream detection when used by plugins

Notes
With the release of the game ratchet & clank: rift apart which used direct storage's gdeflate compression within its game files, I thought... hm, why not just add a scanner for these streams and so I did after investigating the stream structure (was easy to figure out actually). Use via -mgdeflate.

Will games use direct storage? I don't know but if they do then at least xtool has support for the streams used in the games, I just added it so that repackers don't flood me with DMs when they have troubles repacking games, maybe Starfield might use this...

After several tests, the gpu feature can now be used by the general public, use via -g#

[Jahan1373]

Hello
I have been told that you are very forgiving
Can you recommend a good compressor?
Easy and simple work, high compression and works with all site installations, thank you

[Masquerade]

^^
This question has been asked hundreds of times.

7-Zip.

[Jahan1373]

Originally Posted by Masquerade

^^
This question has been asked hundreds of times.

7-Zip.

I don't know that Fitgirl put all these compressions with 7zip program here

[Masquerade]

^^
Obviously FitGirl doesn't use 7-Zip. But you asked for:

Easy and simple work, high compression and works with all site installations

So, 7-Zip is your answer. If that isn't easy enough for you, try built in Windows zip maker

[L33THAK0R]

Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?