Razor12911,
I did use zlib+preflate, I simply forgot about reflate.

After using relate, everything works fine.

Update available

Changes

- updated library support
- updated command line parser
- included x86 build
- fixed depthing issues

Notes

I have added 32-bit build as requested but you'll have to use x86 libraries or find a matching pair of x86/x64 libraries. You can get these from github of a project under the releases section.
The versioning is brought back to the format of 1.0.0 instead of the build number as requested.

@Razor12911

Thanks for this new method... incredible!

On the sample I sent to you yesterday.

@Razor12911, thanks for the 32-bit version:eek:, it will be very useful to include in CIU.
1) Also like to inform you that it is being detected as a false positive by the KIS anti-virus (I don't know if there is anything that can be done).
P.S: The 64-bit version has not been captured.
https://i.imgur.com/rdaSySe.png
2) You could share libraries compatible with the 32-bit version to be fully compatible and avoid errors due to incompatibilities.
3) I wonder if I have 3 games (collection) and each game uses XTool with different methods to compress.
Assuming that only the libraries needed for compression (nothing more than necessary) for each Data.bin file are together with XTool.exe at the time of compressing.
To perform the decompression, can I have other libraries that were not used to compress the Data.bin file together with XTool (Like: Have 1 library next to XTool when compressing and 5 when decompressing)?
Thanks!

1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/...296d/detection
x64: https://www.virustotal.com/gui/file/...7dfe/detection

the code is actually the same so I don't know... :D

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
Attachment 28717

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.

1) For me it displays "Item not found" message in the virustotal links of your post.
Thank you for the informations.
The false positive does not happen when using Windows Defender. I don't understand why KIS is capturing him. (Just because it's x86 (same source code)... It's like COVID, it reaches the weakest organisms (x86)).

2) I know that few use x86 systems. But because CIU still works on x86 systems it is useful to have x86 compatibility.
Maybe in some time, by default, in the CIU, the "ArchitecturesInstallIn64BitMode=x64" and "ArchitecturesAllowed=x64" configs would limit the CIU to only 64-bit systems so only 64-bit versions of compressors would be needed.
But for now, if possible, I prefer to maintain compatibility.
I know that some plugins/libraries only exist 64-bit versions like the Kraken method that requires oo2core_#_win64.dll (I think there are no 64-bit versions of these libraries).

3) I see in your examples that zlibwapi.dll is always with XTool.
Is this library necessary to be together with XTool in methods other than zlib?
I ask why I want to keep the libraries that it is not mandatory to always be with XTool in a subfolder and depending on the method selected for compression before compressing, a copy of these libraries will be made to XTool. After compressing, these copied libraries will be deleted. In the installer decompressors there will be all libraries used with XTool to compress all Data # .bin that will be extracted by the installer. So I asked the previous question (already answered), if I could have other libraries that were not used in the compression method with XTool when performing the extraction.

Thanks for the great job.

1) Yeah a bit unfortunate
2) There are 32-bit versions just that the games usually never ship with them as they are unused and they are 64-bit anyways.
3) As you say "examples", it's just an example that just happen to require zlibwapi.dll if you don't use the method zlib you are not required to include the library

@Razor12911
Does XTool 2020 already support the "lz4" and "zstd" methods? Or do you have any limitations for these 2 methods?
If so, is this usage correct in arc.ini?
@Razor12911, See your inbox.
I will send you a PM with some results using XTool with some samples that KaktoR sent me (Some tests did not inflate).

Update Available

Changes

- improved depthing
- updated library support
- fixed zstd codec issues
- removed fast memory

@Cesar82

-mzstd should work now, as for -mlz4. A plugin for that specific game or game engine should be made, the community can send in samples for games and I'll take a look at them if I can add support or not.

Update Available

Changes

- updated lz4 codec
- updated library support

Thanks for all these informations. Very usefull to us ;)

Update available

Changes

- fixed bug depthing (thanks dixen)

Update available

Changes

- updated oodle codec (fixed lzna bug)
- added custom method configuration

Notes

Custom method configuration is that xtool.ini file near the executable where you can put all the method which can't be passed directly via Freearc when using {option} feature in arc.ini.

xtool.ini
if you want to compress borderlands for example, the command line via FA should look like this

xtool will check if the codec exist somewhere in the configuration then if it does it will convert it "-mborderlands3" to whatever is written in the configuration.

This feature was added to help Cesar's DiskSpan GUI project but it can have several uses. If methods become too many, users can add all the methods in xtool.ini so they know what they are for and use them by referring to them by their stored names.

@Razor12911, thanks for the new feature.
It will be very useful to include the methods of the plugins, but for the ue4 plugin it is necessary to send the key and parameter -m# because the decryption key can be used for any other game that is not listed.
There will be some changes to the "fcnd" plugin that KaktoR sent me, or it is finished (it was huge)?

You can always use SetIniString('CustomMethod','ue4customkey','-mzlib+ue4:m1:some key of a game that is not listed','xtool.ini');, basically you write the key the user has provided and each time use it as -mue4customkey.

as for fcnd, a few things need to be fixed.

Yes I thought about it (setinistring), but I would have to completely separate the command line for the unreal plugin.
See in the VIDEO how it works on DiskSpan_GUI. It works the same way for ue4dt.

The Arc.ini file was fixed with Zlib.
It may be possible to check the method (s) prior to ue4dt and if it is an XTool 2020 method it could be cut (the previous ones together) and sent to XTool.ini and replaced with the name of the key (as an example ue4method=).

I'll think of something later, but this feature of XTool.ini is very useful.
Should I always use the name XTool.ini or if I rename XTool.exe to XTool_x64.exe should I also rename the INI?

Great work that makes me thrill enough to keep up the good work. If you want to make your brand popular just take service from the Go2Top Panel.

Update available

Changes

- updated oodle codec (fixed more lzna bugs)

Is there a way to cap memory usage by xtool?

I ask because I have wasted about 5 hours with Project Cars 3 Kraken codec, where it would get to nearly finished precompressing, then shoot up in memory usage completely crashing my PC needing force restart from the reset button on my case. (Bear in mind this is with xtool capped at only 6 threads of CPU, so memory usage isn't too bad anyway). Oodlerec is terribly slow.

Same thing happens with American/Euro Truck Simulator, in those cases I have no choice to use a chunk size of 8mb otherwise my PC just crashes.

Thanks!

Edit: I get past this just by swapping STDIO for $$arcdatafile$$.tmp $$arcpackedfile$$.tmp - seems good!

Update available

Changes

- updated library structure

Update available

Changes

- fixed command line parser bug
- updated library support

Hi thanks for the great work.

With xtool0.12 I have

On a testing file set mostly zip and 7z files, it could precomp it well
Then I could add srep+FL2 to compress it with a good ratio.

With the recent release xtool 2020 from xtool_0.3.8.7z,

with a setting of

It will not do the similar precomp process, output is

Do I have to adjust/add the parameter to have the similar results?

Edit:

I guess the default values for zlib might have changed. And on some aspects the new version is performing much more like a normal average setting.


For a set of .dat files 800MB, tried to compressing with arc

srep+FL2 compressed it to 414300KB, takes 32 seconds.

xtool_0.12 e:precomp:t75p,c128m:zlib+srep+FL2 compressed to 412000KB takes a big 55 seconds. the zlib codec here is working very hard on something and found a little bit to more to compress further. But it added 23 seconds up on (srep+FL2)

xtool_0.12 e:precomp:t75p,c128m:zstd+srep+FL2 got 414000KB takes 36 seconds, it adds 4 seconds up on (srep+FL2)

xtool_0.3.8 e:precomp:t75p,c128m:zlib+srep+FL2 got to 414000KB takes 33 seconds, it adds 1 seconds up on (srep+FL2)

xtool_0.3.8 e:precomp:t75p,c128m:zstd+srep+FL2 got to 414000KB takes 33 seconds , it adds 1 seconds up on (srep+FL2)

Update available

Changes

- fixed future stream bug

@github

The command line has changed, check the example to see how it's used.

Furthermore, the old xtool used reflate without the knowledge of the user. The new one doesn't so if you have a scenario where the old xtool gives better results, then you may need to combine zlib with reflate or preflate.

Wdl
 

small test on WDL

Thank you. So with xtool 2020 I should use zlib+reflate -d or zlib+preflate -d
on command line with this it does detect many streams

how to I pass the -d level in the arc.ini?
does xtool0.12/2020 support the writing of {options} in arc.ini?

I tried

Those doesn't make any difference to the one without d3 option

are you precompressing something that requires depth level to be set to 3 or higher? as for choosing between reflate or preflate, you need to know how each works as they have their advantages and disadvantages.

which should explain their differences seen here. What I would point out as the main difference, reflate supports streams with no tail while preflate doesn't so if a special deflate encoder was used or if a stream had been cut off, preflate will be unable to process such streams but reflate is able to handle these.

as for {option} via freearc, I'd first say you need to understand the command line syntax of xtool. The old xtool and new xtool use different syntax.

Ideally this would become

then used as -mxtool:mzlib:c32mb:t100p from freearc.

Thank you for the example.

I have it setup up like this


However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

This true.

I wonder though, A depth of 3-5. I asked before if there is such data you have on hand because this is basically the same as having data, compress it using zlib, the output, compress that using zlib, do this at least 5 times. That's what depth is.

A depth of 1-2 is usually enough on everyday files such as pictures and documents as these are usually shipped in a zip file, that zip file is compressed using deflate and if it contains pdfs or png images then, that's when you need to use depth otherwise, don't touch it as you'll be increasing precompression time unnecessarily and have no gains in ratio.

Hello Razor, is there anything that can be done to cap the memory usage of xtool at a certain amount?

I am aware of a cmem variable in arc.ini packcmd (only small knowledge) however I was wondering if there's any other ways about this.

I find on larger workloads/workloads with high amounts of streams, xtool fills up my RAM and completely crashes my PC.

I am working with 16GB memory. Is there anything that can be done without changing chunk size?

Currently I'm capping threads and using $$arc$$ instead of stdio.

-lm

:D:D thank you for implementing such a feature, I thought it got removed when the project was started (I remember it being in 2019 version).

Thanks again.

Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method

you can't cap the memory usage of the program, you can only reduce it. When you are compressing, usually each thread gets its own chunk. Using -lm parameter makes all thread use 1 chunk. As for memory usage, that's highly dependent on the user and the data they are precompressing, if you set very high chunk sizes, be prepared for high memory usage. When decompressing, xtool tries to not use more than 512 MB ram automatically unless if there are large streams or if the recompressing library requires more memory.

did quick test on mad max with/out --dedup=xtool.bin to se mem usage...
with --dedup size to 55gb
without --dedup size 57.1gb
but memory dec the same around 6gb.
*note only test with mzlib+srep

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib -c128mb -t12 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>


[External compressor:srep]
header = 0
packcmd = srep.exe -m3f -d1g -a2 $$arcdatafile$$.tmp - <stdout>

Mad Max is a bad example for dedup testing. They use really huge deflate chunks with level 1 compression, so the amount of dupe chunks is not very big, unlike the duplicated data in those chunks when decompressed, that's why srep is good solution here.

so game like (just cause 3 &4, ac syndicate...) using dedup xtool will give similar results as mad max :(
so i guess dedup xtool most useful on small chunks :confused:

For future repacks Borderlands 3))

pakchunk0-WindowsNoEditor_0_P.pak

packcmd = xtool.exe precomp -mzlib+preflate -c128mb -t100p-1 - - <stdin> <stdout>

packcmd = xtool.exe precomp -mzlib+preflate -d3 -c128mb -t100p-1 - - <stdin> <stdout>

Final compress with srep+lolz

With -d3 - 61 mb
Without -d3 - 270 mb



NOTE.
Before precompress *.pak is decrypted with ue4dt
Used XTool v0.3.9

For some reason, they gave me an decompression failed corrupted archive error by decrypting the pak file.

unarc.dll returned -11

I put a config on Bunti's Windows Phone Bug Free Installer arc.ini file.

Is there anything wrong?

Solved by adding a new resource.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+ue4:m1:kC40250B917281AC9874D93C53E429255177E A6E038A117054A1F1C5DA92AB26A -c128mb -t75p -d3 --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

Hi. Someone tested xtool 0.3.9 with the command "-mlz4" with some game and it worked? I'm testing it with several games made with unity engine and it doesn't work. Tested also with all the downloadable lz4 dll (i have downloaded from github versions 1.7.3, 1.7.4, 1.7.4-2, 1.8.0, 1.8.1.2, 1.8.2, 1.8.3, 1.9.3). Looks like xtool doesn't try either to precompress the files, because it is very fast, maybe it is ignoring that command "-mlz4"? Is that command a placeholder or what?

Maybe, try using the old XTool?

Some use LZMA, but some use LZ4?

Seems like razor was planning on it?