@Razor12911, can the new -sm# parameter be used from the freearc command line using comma "," to separate srep parametters?
I would also like to know if running xtool with the -g90p parameter on a system without a dedicated video card will also work or could it crash the system?

Thanks!

Update available

Changes

- added footer feature in configuration plugins
- added thread priority controller via -T# parameter

Notes

Footer detection has been added to configuration which means if there's no way of knowing the size of a stream but if within the stream itself has a footer or end of file/stream marker, you can input that as a signature for xtool to look for which will help you in determining the size of the stream.

For example, jpeg image file has the header 0xE0FFD8FF but if you looked at the last 3 bytes, you'll notice that in all of them, they end with 99 FF D9, this will be the footer and your configuration should look something like this

The offset is -4 because the signature is part of the stream, you have Footer=0xD9FF99 which xtool will search for after finding Signature=0xE0FFD8FF, the compressed size is then Footer + 7 because the signature + the footer are both part of the stream.

Another use case for this is if you have streams stored in sequence, if their signature is 59 61 7A 30 and the next stream has the same signature, you can write the next stream as the "footer" of the current stream.

This time, the compressed size doesn't added both signature and footer because the footer is actually the header of the next stream, there's one caveat with doing it this way however, the last stream will not be captured.

Thread priority option is added and can be used via -T and the values are 0..6 with zero being idle and 6 being real-time (time critical), if you have laggy mouse while installing a game knowing full well that you have set 100p for xtool and have lolz or lzma working and toasting your CPU and saying it's xtool's problem then you should use this but know that setting this too high means your system will be unresponsive or setting too low making installation longer, well... up to you. Too many notes...

@Cesar82

You can use commas to separate parameters.

For GPU, xtool performs a check to see if the system has a GPU with at least 1GB VRAM and even then it makes sure that there's at least VRAM 512MB left unused. If the condition is not met, it doesn't use GPU at all.

Update available

Changes

- Minor bug fixes

Notes

This release contains recompiled zlibwapi.dll, preflate_dll.dll, lzo2.dll (to remove Visual C++ 2010 requirement) and xdelta3_dll.dll simply because there are people who are using Windows XP in this day and age, shocking I know...

Damn. I'm lame :D

How can i use this with XTool076 for JPG stream compression?

[Stream1]
Name=jpg
Codec=brunsli
BigEndian=0
Signature=0xE0FFD8FF
Footer=0xD9FF99
Structure=Signature(4),Stream,Footer(3)
StreamOffset=-4
CompressedSize=Footer+7
DecompressedSize=0

Solved :
My sample JPG have different header.

Thank you Razor12911 for your updated xtool and libraries...anyway, tested unpacking speed of your last three x64 zlibwapi.dll library compiles, multiple times (did not test x86 version at all)...

Unpacking speed of zlibwapi.dll 1.2.11/2020-07-18/535552 bytes is _always_ faster than any of your other, newer, 1.2.13 compiles (2022-12-31/544768 bytes and 2023-07-11/512000 bytes)
Newest XP compatible zlibwapi.dll is by far slowest during unpacking, probably because it does not use some of advanced CPU instructions available after XP. Anyway, I truly welcome XP support because I consider games that are playable on XP should be able to unpack on XP...but if making .dlls XP compatible comes with obligatory speed loss than maybe it is better to provide separate XP compiled libs, just my thoughts of course. Ideally, one compile should work on all OSes with max speeds but don't really sure if this is possible...

Following are examples of decompression speed of _same_ arc packed with xtool:zlib+srep:m3f+lolz ,all that is changed is zlibwapi.dll before unpacking (encoding speed is not primary important to me, arc is packed using one of mentioned zlibwapi.dll but this does not impact decompression speed, tested)
Tested on Windows 11, 20 threads CPU, xtool 0.7.4 used, -t100p in arc.ini
Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it..

Regards

infovs: Please testing more version older dlls...

Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.

This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.

I tested all zlibwapi.dll versions that I could find since ztool..Razor12911 version from 2020-07-18 (535552 bytes) was the fastest at decompression. I also did my own compile of zlibwapi.dll 1.2.11 with MS VC 2010 and also 1.2.13 with MS VC 2010/2015/2022 -> VC 2022 1.2.13 compile produced best result but Razor12911 version was still fastest at unpacking, although not by much, speeds were pretty close. Actually only this latest 2023-07-11/512000 bytes version speed drop was noticeable for me, that's why I posted my observation.
Also, if you use MS VC and want your compile to be XP compatible you are pretty much forced to use VC 2010..yes, you can use XP platform toolset on some later MS VC versions but I had some issues with it. Never compiled with MinGW though so I can't comment on that.

I didn't do anything to 1.2.11 to make it fast, 1.2.13 itself is the one that is slow as you can see with the benchmarks, winxp compile has little to no impact on performance.

There's an issue opened on github regards to the speed which can be found here.
https://github.com/madler/zlib/issues/820

There was also a security vulnerability discovered in 1.2.11 and I don't know if patching this issue is what made zlib slower or not moving forward.
https://www.cybersecurity-help.cz/vdb/gnu/zlib/1.2.11/

The beauty of xtool is that the user can swap libraries if they prefer to use their own, there's also zlib-ng (utilizes modern CPU features) benchmarks added, as you can see it is faster however I doubt it produces matching crc as standard zlib but good to know nonetheless. ;)

Update available

Changes

- added universal scanner for DirectStorage gdeflate streams
- added the use of gpu for caching and virtual memory purposes
- updated depthing feature to improve stream detection when used by plugins

Notes
With the release of the game ratchet & clank: rift apart which used direct storage's gdeflate compression within its game files, I thought... hm, why not just add a scanner for these streams and so I did after investigating the stream structure (was easy to figure out actually). Use via -mgdeflate.

Will games use direct storage? I don't know but if they do then at least xtool has support for the streams used in the games, I just added it so that repackers don't flood me with DMs when they have troubles repacking games, maybe Starfield might use this...

After several tests, the gpu feature can now be used by the general public, use via -g#

Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?

xtool erase <folder> <archive> <database>

Update available

Changes

- fixed an issue where xtool would crash on Windows XP and other systems upon closure
- memory optimisations
- improvements with database "based" plugins

Thank you Razor12911, also..is there a reason not mentioning there is also new CLS version of xtool in your 0.7.8 update? Not stable/recommended yet?

^^
Run some tests and see what you find. That's what I'm gonna do when I return to my PC.

Please note that FreeArc is an x86 program, so in order to use the x64 CLS variant, you will first need to use some kind of wrapper such as wrapcls by 78372 or FAZip (not FAZip32).

It's gonna be a lot of fun to play with. A cls is what I've been secretly wishing for, thanks Razor :D

@Razor :

Maybe an XTool 0.7.8 bug ,using FLAC. Why is that increased memory consumption ?

Compression : -mflac:l8:c128mb , 1 thd ,approx. 800 MB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

Compression : -mflac:l8:c128mb , 4 thds ,approx. 1.8 GB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

All decompression settings used % to set the number of threads. Using T4 instead of t100p cannot decompress the archive :
ERROR: general (de)compression error in flac:l8:c128mb

Maybe other cases affected. Dont know.

i5-6600

There's two different code sets for decoding in xtool, one is for multi threading which caters for speed at a cost of increased memory usage because the streams have to be reorganised before they are written to disk and as such, high memory usage because they are stored in memory first. The other code is when you have set 1 thread of if a block has 1 stream in it, for this as soon as a stream is processed, it is written to disk hence why the reduced memory usage in -t1.

For flac, I'd suggest using -t1 for decoding simply because flac is decompressing when installing unlike other codecs where they require multi threading to be fast as they are doing the opposite (compressing) during installation.

A small test on some audio files

Using 12 threads, used 1.23 GB ram

Using 1 thread, used 185 MB ram.

The single threaded speed is reasonable while using 12x number of threads only gave 2.2x speed benefits while using nearly 7x memory. Obviously due to IO bottleneck show that it's better to just use 1 thread.

Hi Razor12911, I know similar question has been posted before but can you please re-consider renaming hardcoded srep.exe call in x64 xtool version to original, srep64.exe ? Since you made xtool available for both x86 and x64 versions I started using both when appropriate (making XP compatible installers for games that can be played on XP...) and hardcoded srep.exe is giving me headaches because both x86 and x64 are executing srep.exe. x64 xtool version should be executing srep64.exe IMHO :o
Anyway, thanks again for your xtool, best regards

Update available

Changes

- added recompress streams feature
- added reassign streams feature
- added dummy codec
- added data transfer buffer for srep when dd# is used
- added memory checks to ensure memory usage does not run wild
- exectuable plugins (via xtool.ini) are no longer required when decoding if they were never used
- internal stream deduplication now reports speed and memory usage benefits
- configuration based plugins can now have multiple names (if multiple games use the same configuration)
- fixed xtool crash if an incorect path for plugins was provided
- fixed lz4/lz4hc codec bug when used directly without any plugin
- fixed an issue where using fast-lzma2 compression would crash when decoding
- fixed an issue in DirectStorage gdeflate codec
- fixed an issue with execute command in stdio mode
- deduplication memory requirements improved
- memory optimisations
- reduced memory requirements for large streams that require patching
- removed ability to inject libraries to exe (buggy)
- removed patch command (uses too much memory)
- removed archive command (no one uses it)
- updated oodle scanner
- updated lz4/lz4hc universal scanner
- updated lz4/lz4hc/lz4f codecs
- updated zlib codec
- updated zstd codec
- replaced gpu caching feature with normal system memory cache (it doesn't work well on AMD gpus), use -p#
- replaced xdelta3 with zstd patching engine
- srep64 executable considered in x64 build of xtool

Notes

First and foremost, huge thanks to KaktoR for helping with this update, he was full of energy when we started but I think I broke him towards the end...
Attachment 35375

This update was supposed to be released a long time ago, but he would find more bugs hence the long list of changes.

I won't say much other than notable changes, xdelta was replaced by zstd patching engine (yes... zstd can make patches too, shocking I know...), reason for replacement xdelta gives me less options coding wise to work with in terms of memory management.

Injecting libraries feature also removed as some libraries when injected completely stop working or worse, cause xtool to crash.

I've quietly added cls plugin in the previous update and did not tell anyone about it simply because it was never tested, for this update It was tested and it works so long as you never used fast-lzma2 in any of the archives when using it with installations created with inno setup as it would simply cause a crash.

I've left 0.7.8 on the main post in case there are issues with this update even after extensive testing.

Very embarrassed to ask such a simple question regarding database creation, but I can't seem to wrap my head around it. I've got some assets I'm attempting to rip some assets that are compressed inside an archive using deflate.

To do this I'm pretty confident in saying xtool just needs a database to reference, with the command looking like

as a user pointed out to me a while ago.

Where I get stuck is creating the database, the title in question is "The Wanderer" a UE4 title (unencrypted) with a singular ~30GB ".pak" archive containing said assets, I've extracted the assets I'm interested in ripping to their own folder and have so far tried using xtool 0.7.9's GUI to generate the database since I'm not too confident the syntax I'm using is correct when running xtool from the command line.

Here is how I've setup the database creation though I'll admit I'm unsure of whether I've got the "Input" and "Source" correct.

https://i.imgur.com/2IEnbQ7.png

However all I get from this configuration is the following message, which hangs for a few minutes despite some disk activity as per task manager, and then indicates the process is completed however no database file is created.

https://i.imgur.com/az0bW9p.png

Very unsure and confused about what parts I'm mucking up or whether XTool just has a limitation on the size of inputs that can be processed.

Don't use the GUI. In that screenshot, you are using the wrong database generator and your inputs+outputs need to be reversed.

And you cannot rip compressed data. It must appear in plaintext in order to be erased by xtool.

Someone knows how to use rz compressor with xtool like xbcm?

Edit: thanks to Razor12911, now xrz has been added to XTool!

I can help if you want?
 

Now I got some more experience with compression, I am always open to help if you want.:)
But I totally get it if not! So don't worry

Hi guys, I like to take this opportunity to say hello to old contacts who have known me here on Forum since 2011, and the new guys who are passionate about compressing and installing games. I'm glad that the forum is still active, even if not as I remembered...:)
A big hello to friend Zee, who still works flawlessly with his work to this day.
I've been away a lot for work, so I'm quite rusty on the subject.:)
I wanted to ask someone here in the thread if they had managed to implement the added function of srep in xtool, or at least if I understood correctly if could be done and how....
Added in last xtool "- srep64 executable considered in x64 build of xtool" - //- "-sp# - srep parameters (separate params with ":")". thankss

I have archived XTool 2020's release history, related tools and plugins to a github repository in case of another FileForums outage. This repository also contains collections of oodle, lz4 and zstd libraries for precompression and decoding.

https://github.com/Masquerade64/XTool_2020_Archive

I have also archived Razor12911's legacy tools into a separate repository as they may still be desired:

https://github.com/Masquerade64/RazorTools_Archive

-------------------------------------------------------------------

For those who don't know, XTool progress has been moved to Patreon
https://www.patreon.com/xtool

Consider joining to help support Razor12911.

-------------------------------------------------------------------

Latest Release: XTool v0.8.1 (6/12/23)

https://www.patreon.com/posts/release-v0-8-1-94158307

Help me
 

Hello
I have been told that you are very forgiving
Can you recommend a good compressor?
Easy and simple work, high compression and works with all site installations, thank you

^^
This question has been asked hundreds of times.

7-Zip.

I don't know that Fitgirl put all these compressions with 7zip program here

^^
Obviously FitGirl doesn't use 7-Zip. But you asked for:

So, 7-Zip is your answer. If that isn't easy enough for you, try built in Windows zip maker :p

New Release: XTool v0.8.2 (11/1/24)

https://www.patreon.com/posts/release-v0-8-2-96291433

New Release: XTool v0.8.3 (16/1/24)

https://www.patreon.com/posts/release-v0-8-3-96616797

New Release: XTool v0.8.4 (25/1/24)
https://www.patreon.com/posts/release-v0-8-4-97128362
You can always support Razor for his work on his patreon

New Release: XTool v0.8.5 (03/2/24)
Mermaid test
https://www.patreon.com/posts/release-v0-8-5-97747742
You can always support Razor for his work on his patreon

New Release: XTool v0.8.6 (15/2/24)

New Release: XTool v0.8.7 (22/2/24)

Support Razor12911 on Patreon

Is there an issue with LZ4 stream detection in recent versions of XTool?

The LZ4 streams in the following sample file are correctly detected in old XTool 012 but not in recent versions...or am I missing something?

File sample.pak

XTool 012

xtool.exe e:precomp:c16mb:lz4 sample.pak sample.pak.xt

9,327,964 => 13,522,962

XTool 0.8.7

xtool.exe precomp -mlz4 sample.pak sample.pak.xt

It just seems to get stuck in a loop forever.

Here is the link to sample.pak -:

https://drive.google.com/file/d/1Yrj...ew?usp=sharing

You cannot use lz4 directly. Plugin is required always if you want to use lz4/lz4hc.

However your sample seems to be compressed with lz4hc, because recompressed stream is bigger then original stream.

I don't know from what game this sample is. Either make a plugin configuration (ini file) or search for bms script and use bms2xtl with correct settings.

Yes, you're right - new XTool seems to detect some of the streams as LZ4HC. Not sure why the old XTool works perfectly when choosing generic LZ4 though.

The file is from CoD:MW Remast btw. All of the scene repacks are horribly out of date so I was compressing it myself.

In this case you have to make a configuration plugin in order to precompress this game with newer xtool versions, as I am pretty sure there is no bms script for this game.

Application cls-srep.dll instead of srep.exe is it possible to implement this when using -dd#?

^^
Nope. cls-srep_x64.exe works differently to a typical srep.exe. It's focus is decompression only and uses different parameters afaik.