XTool 2020 (Main Project)

[Ele]

Mr. Razor
Is lz4, lzo and zstd still not supported for xtool 2020?

[elit]

Razor, pls check my above post No. 243 and consider that idea for parameters parsing to finalize specification. It would make xtool compatible with FA including GUI.
Finally, please consider including all parameters list with their description within command line help instead of external file.

EDIT: Also, provide full documentation on plugin system reserved keywords and their exact meaning(like if offset jump is meant from current position or beginning, if reading string advance position or not and so on), their full list etc.

Thank you for everything.

[Cesar82]

@Razor12911, is there any change from XTool 2020 to have a 32 bit version too?
Perhaps in the thin version l of the XTool 2020, it would be great.
Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it, having to choose to use the old XTool.
Also if the user has a 32-bit system installed he will not be able to use Xtool, even if it is for a game that will be installed on another system in the future.
Thanks!

[elit]

Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it
Why? What would happen, according to you? 64/32bit xtool have absolutely no relation to 64/32bit games. Only libraries used in xtool, hence why crilayla was dropped.

[Ele]

Why? What would happen, according to you? 64/32bit xtool have absolutely no relation to 64/32bit games. Only libraries used in xtool, hence why crilayla was dropped.

I think people with 32-bit operating systems use 32-bit versions of games. If they want to back up their games, they must use the 32-bit compression tool. Yes, xtool's architecture (32 or 64) is for the operating system only and not for the game.

[Cesar82]

Why? What would happen, according to you? 64/32bit xtool have absolutely no relation to 64/32bit games. Only libraries used in xtool, hence why crilayla was dropped.

f you create a compressed Data.bin file using 64-bit XTool we will not have to extract (install) on 32-bit systems.

The question was addressed to Razor12911. Most of the time other users are commenting and giving their answers and it turns out that the question post is left behind and the user who was asked many times ends up not seeing the question.
Please, other users, avoid answering questions that have a username mentioned, so it is easier to be answered. Thanks.

[Razor12911]

You may want to be interested in this.

I was trying to inflate cpk file from Diablo 3 from nintendo switch. Normaly these are crilayla but gfs detected lz. So I tried old ztool, some older xtool(date modified says 26.april 2019) and xtool2009R3.

For reference I tried single file "Act1.cpk" of 814mb size. GFS detected potential of inflation of up to ~1800mb.

Ztool could not inflate past few 10's of mb if any, regardless of setting like m2, m3 etc.

Later xtool2009R3 depended on codec, most were at about 100mb extra only, except reflate which did go to 1.1gb. However final packed size after srep+xlzma was about same which make me think of possible huge overheat?

Finally, that older xtool with :high setting was able to inflate to.. 1.7gb!!
I tried to compress that and got about ~40mb better compression.

General final compression after codecs was around 740mb+, this one went to ~704mb. I wonder if this was real inflated data to 1.7gb or too much overheat. I must also say that older xtool was able to do it only with :high option which unlike ztool is not even know or documented and probably unofficial. Also it took very long time compare to say ztool:high, it seems to work differently.

I have uploaded the file act1.cpk for further research if you are interested:
https://megaup.net/1zEg1/Act1.cpk

"File not found"

btw, remember how this xtool works. Everything is decided by the user, before when you used zlib in the older xtool and ztool, if reflate library were nearby, it would use them so perhaps you need to combine zlib and reflate to get the desired output.

Having a few problems decompressing with the fcp option, I have tried a few different unpackcmds:

[External compressor: fcp]
header = 0
unpackcmd = xtool decode --dbase=fcp.xtl - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode -mfcp - - <stdin> <stdout>

Yes, fcp.xtl and liblz4.dll are with xtool.exe

Errors like:

G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed 0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed 0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed 0%
ERROR: write error (disk full?) in compression algorithm fcp

This is just on 1 file with no extra compressors.

Is there something wrong with my unpackcmd?

All issues related to libdunia will be fixed via a separate plugin of xtool

In fact you already completed the work, why abandon it? So for the love of god I beg you, put it back in latest version!

PS(Zstd and lz4 are temporary and will stop working after time because of their retarded design. Not sure about lzo(that is actually a question, why is standard lzo not working in dunia and unreal engines and need special treatment? I thought its a stable design like zlib?). But crilayla, lzss, yaz0 and such should remain compatible.

I temporarily removed crilayla because I wanted to have a reworked 64-bit version via plugin support, not many games use it so it shouldn't ship with xtool. Oodle, zlib, lz4 and etc, these algorithms are common therefore they ship with xtool, anything else will be a plugin.

-grittibanzli inflate completely randomly. Sometimes tested 20mb inflated zip file is 34mb, other times 31.6mb, another time 33mb atd. exactly same setting and just repeating same command on console.

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)

-preflate and reflate both did to same ~27mb, but preflate took ~6sec while reflate did it in ~1sec.

Looks like good old reflate is still the best zlib codec to stick.


EDIT(and ztool is also able to inflate to 27mb without :high or :m2 !) <<UPDATE(BS, scrap that claim, need m2 or high indeed)
EDIT2(ok I checked manual again and learned about zlib+reflate combination benefits. Maybe this is best way. It took ~3sec compared to ~1sec on deflate alone though. Not sure how big those benefits are in real application yet.)
EDIT3(have to explicitly state 'mb' in -c option instead of just 'm', e.g. -c128m vs -c128mb. No big deal but used to ztool, was more flexible. PS: I am really liking the idea of zlib+reflate in one go, before I used to pass twice in test to find out which one to use.)

zlib is the first codec that tries to process streams, if you mixed the methods zlib+reflate... if zlib fails, it gives it to reflate to try processing it.

Mr. Razor
Is lz4, lzo and zstd still not supported for xtool 2020?

lz4 and zstd are in xtool but lz4 is just a placeholder for now, lzo will be added as a placeholder too.

[Razor12911]

Regarding '+' and ',' characters issues, why not just use ':' for everything and dissect the parameters in the code instead? I mean, for example:
xtool -mzlib:c32mb:reflate:l9^This is perfectly easy to scan because names like zlib, kraken, reflate etc are unique and cannot appear as codec flags. So in your code you would just look for a word that represent a codec in the list and then all other params would be flags until next codec name is detected. So for example:
-mzlib:kraken:{param} program would still know that zlib is with default parameters and kraken with whatever you put behind it simply because it knows about reserved words.

Also regarding versioning, whats wrong with simple v1.032, v1.033, v1.034 or anything like that?

Just a suggestions of course, I don't want to feel ungrateful or anything.

regarding character parsing, I could change it but the thing is, xtool doesn't really know reserved words because there are none but I can change it if that's what people want.

The reason I went with a versioning like this is because the there are already 3 different xtools so to not confuse people, they should just check the YYMM it was released on as the old xtool, 0.0.19... has special editions from the request of repackers so basically the versioning is so confusing so I then decided I'd only adopt it once I think the project is a guaranteed success. 2019 is an example because what I tried to do fail, so I tried again with 2020 to see if it will work with little hope of success but as soon as it seems like the idea of external plugins is possible and can be done without bugs then versions will be brought back.

[Razor12911]

@Razor12911, is there any change from XTool 2020 to have a 32 bit version too?
Perhaps in the thin version l of the XTool 2020, it would be great.
Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it, having to choose to use the old XTool.
Also if the user has a 32-bit system installed he will not be able to use Xtool, even if it is for a game that will be installed on another system in the future.
Thanks!

I don't know if I said it here on the forum or to someone who asked for 32-bit version of xtool. I stopped shipping 32-bit xtool because there are certain mistakes that some people make when they use xtool. An example is when they use oodle/lz4. The library you use must be compiled from the same source code, this you must make sure (if the 64-bit dll of lz4 is v191, make sure the 32-bit dll is also v191). So what happens is people use 64-bit version of the library and then ship the 32-bit version of xtool along with different libraries making xtool to cause CRC errors because the libraries didn't come from the same source. 32-bit is available on request but it will be the users fault if they have issues with it. What you also must remember is xtool ships with xdelta, some streams cannot be restored so it tries to salvage a few streams but if there is a mismatch in the data caused by different dll, it causes xtool/xdelta to return errors.

The question was addressed to Razor12911. Most of the time other users are commenting and giving their answers and it turns out that the question post is left behind and the user who was asked many times ends up not seeing the question.
Please, other users, avoid answering questions that have a username mentioned, so it is easier to be answered. Thanks.

Relax I backtrack posts when I return and answer all of them when I have time, I did say that I spend less time here and sometimes what happens is if a question is directed at someone, if they are not around and other people don't step in, it still falls behind and the question is still never seen (especially with threads). So I think he did what was best by giving a response right away best he can according to his knowledge.

[Cesar82]

I don't know if I said it here on the forum or to someone who asked for 32-bit version of xtool. I stopped shipping 32-bit xtool because there are certain mistakes that some people make when they use xtool. An example is when they use oodle/lz4. The library you use must be compiled from the same source code, this you must make sure (if the 64-bit dll of lz4 is v191, make sure the 32-bit dll is also v191). So what happens is people use 64-bit version of the library and then ship the 32-bit version of xtool along with different libraries making xtool to cause CRC errors because the libraries didn't come from the same source. 32-bit is available on request but it will be the users fault if they have issues with it. What you also must remember is xtool ships with xdelta, some streams cannot be restored so it tries to salvage a few streams but if there is a mismatch in the data caused by different dll, it causes xtool/xdelta to return errors.



Relax I backtrack posts when I return and answer all of them when I have time, I did say that I spend less time here and sometimes what happens is if a question is directed at someone, if they are not around and other people don't step in, it still falls behind and the question is still never seen (especially with threads). So I think he did what was best by giving a response right away best he can according to his knowledge.
Thanks for the quick response.

If I understand correctly, users compress a game with a library version and try to extract it with another.
But can't this happen with only the 64-bit version of XTool (Compressing with XTool X and trying to extract with XTool version Y)?
You always share the libraries together with XTool, so you should logically use them to decompress.

My idea was to update XTool on CIU so I decided to ask.
Now in CIU the libraries and decompressors are no longer included in the setup.exe, being added when the game is deleted in an external file, thus avoiding a bit of using different versions of the DLL and exes for extraction.

You said always go back on the posts and respond.
You ciu this question referring to ue4dt in the post HERE (https://www.fileforums.com/showpost.php?p=488978&postcount=76).

Can I have specific plug-in files for a game like "oo2ext_7_win64.dll / cp2077.ini" inside the XTool folder when using other methods?
If I want to include several plugins (specific files needed for a particular game) I can put them all in a folder with XTool and in the files for decompression do not include these files, or they must also be together in the extraction, even if it is specific to another method ? Of course, if the libraries in the "Libraries" folder shared with XTool exist alongside XTool in compression, must be together in decompression.

[Razor12911]

Thanks for the quick response.

If I understand correctly, users compress a game with a library version and try to extract it with another.
But can't this happen with only the 64-bit version of XTool (Compressing with XTool X and trying to extract with XTool version Y)?
You always share the libraries together with XTool, so you should logically use them to decompress.

No you don't understand. The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.

You said always go back on the posts and respond.
You ciu this question referring to ue4dt in the post HERE (https://www.fileforums.com/showpost.php?p=488978&postcount=76).

This is user error, someone tried to help you solve this issue in the next post.
https://www.fileforums.com/showpost.php?p=488982&postcount=77
therefore I didn't intervene.
-mue4dt:-m1:-k0x115EE4F8C625C792F37A503308048E79726E512F0BF8D2A D7C4C87BC5947CBA7+xzlib+srep:m3f+4x4:lzma+diskspan :4420mb:4470mb
the problem here is simple, you added -mue4dt:-m1:-k0x...

I don't know if that was the issue or with the program itself.

Can I have specific plug-in files for a game like "oo2ext_7_win64.dll / cp2077.ini" inside the XTool folder when using other methods?
If I want to include several plugins (specific files needed for a particular game) I can put them all in a folder with XTool and in the files for decompression do not include these files, or they must also be together in the extraction, even if it is specific to another method ? Of course, if the libraries in the "Libraries" folder shared with XTool exist alongside XTool in compression, must be together in decompression.
not sure if I follow but you can probably do that, add several plugins. Even if they are not inuse. Or tell users to copy and paste the plugin info (ini) to a static file that you will use.
For example you can just make a custom configuration file, like extmethod.ini in your tools or CIU.
If a user wanted to use a certain plugin, instead of copying cp2077.ini, they can just copy its contents to extmethod.ini. this is also allowed. You can put in as many methods in there as you want but they must follow the standard ini section. [Stream1] ... [Stream2]... and etc. These configuration files as I mentioned are not required for decoding.

Remember, placing oo2ext_7_win64.dll as an example. This means xtool will use that library for everything oodle related so if your question was, can you make a folder which xtool will read cp2077 where there will be (oo2ext_7_win64.dll and cp2077.ini) and another game tc2 which uses (oo2ext_5_win64.dll and tc2.ini), xtool will use oo2ext_5_win64.dll for all of them.

[Cesar82]

No you don't understand. The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.
But you usually share the libraries with XTool.
So if any user is going to replace Xtool with XTool x86 they must also replace the libraries with the libraries that come with XTool x86 (In the file you shared from xtool x86).

the problem here is simple, you added -mue4dt:-m1:-k0x...
I cannot include "-" in the parameter.
So that must be the mistake. I'll check it out and send it to KaktoR to test it, but I think that's what happened. Thanks!
I did not pay attention to the user's Masquerade response. If he had highlighted the "-" he would have noticed.
It's even generating the wrong method on the DiskSpan_GUI that I created...
https://i.imgur.com/1hJDlir.png

not sure if I follow but you can probably do that, add several plugins. Even if they are not inuse. Or tell users to copy and paste the plugin info (ini) to a static file that you will use.
For example you can just make a custom configuration file, like extmethod.ini in your tools or CIU.
If a user wanted to use a certain plugin, instead of copying cp2077.ini, they can just copy its contents to extmethod.ini. this is also allowed. You can put in as many methods in there as you want but they must follow the standard ini section. [Stream1] ... [Stream2]... and etc. These configuration files as I mentioned are not required for decoding.

Remember, placing oo2ext_7_win64.dll as an example. This means xtool will use that library for everything oodle related so if your question was, can you make a folder which xtool will read cp2077 where there will be (oo2ext_7_win64.dll and cp2077.ini) and another game tc2 which uses (oo2ext_5_win64.dll and tc2.ini), xtool will use oo2ext_5_win64.dll for all of them.
So the ideal would be to put just XTool and the common libraries, and put the plugins in separate folders and create copies of these plugins to the XTool folder when initializing the compression and deleting after completion.

Perhaps a configuration file with a path for additional plugins would be a idea.
Example: Use an XTool.ini file and place keys such as Path1=Plugin1, Path2=Plugin2, etc.
When starting XTool, not only load libraries and files that are in the XTool path, but also check and load the subfolders with those paths.
The scan of the keys would have to be checked as long as the key Path#= exists, and not if it is empty because to disable a plugin it would only be necessary to pass an empty value to key path#=. This is only an idea if it doesn't hinder the performance of XTool.

[Ele]

The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.


Razor12911, Thank you for the information.
It's very important to know because I've faced this situation before. It took a long time to find the cause of the CRC error. But I failed.

[infovs]

Razor12911, seems like you broke xtool:kraken with latest version.
I only tested on Death Stranding (of course, decrypted with dst)...you can easily test it yourself with your own dst_r1 package.
Use your original pack.bat -> data.arc is inflated (using oo2reck).
Use xtool_2011_r1 xtool.exe with xtool:kraken -> data.arc is inflated.
Use xtool_2012_r1 xtool.exe with xtool:kraken -> data.arc is NOT inflated.

P.S. Why am I using xtool for this when everybody is using oo2reck? Because xtool:kraken decompress FASTER than oo2reck. I don't care if compression is longer (and it is), decompression speed is more important for me.

[KaktoR]

I guess I have the same. kraken does not work with latest version (and I thought I was just to dumb :p).

[Masquerade]

Is there a progress update on Watch Dogs precompressor, or if not, some guidance on how to write an xtool config in order to create one?

[Razor12911]

Update available

Changes

- added library support
- added compress, decompress, encrypt, decrypt, hash, delta functions (used by library)
- added lzo codec placeholders
- fixed oodle bug
- fixed lz4 bug
- removed libdunia codec

@infovs, KaktoR
Should be fixed now

@Masquerade
Wait for documentation update

To everyone

This is the final major update of xtool, this is a tiresome project and from now onwards it will only be getting minor updates and bug fixes. If you're worried about support for games released in future which xtool has no support of, I have added 3 ways of adding plugins to xtool and that's either by a database, a configuration or a plugin that is written in Delphi or C++ (a well written documentation that explains how these work is underway).

I removed libdunia because this codec is not suppose to be part of xtool, I added it become some scene groups were on the roll in bypassing Denuvo so I thought they have this one in the bag but turns out people have to wait a little longer which gave me time to finish library support. Library support is basically dlls written to add additional support to xtool without updating the main project. Think of it like the CLS of Xtool so be sure to check Plugins thread for the separate plugin.

__________________

I wish everyone a Merry Christmas and a Happy New Year.
This year was terrible and I have high hopes for the next one so see you then. :)

[KaktoR]

oodle is fixed, thanks :) Merry christmas to you too!

from rdr2

Compressed 1 file, 1,018,142,190 => 2,447,518,046 bytes. Ratio 240.39%
Compression time: cpu 0.88 sec/real 111.78 sec = 1%. Speed 9.11 mB/s
All OK

Extracted 1 file, 2,447,518,046 => 1,018,142,190 bytes. Ratio 240.39%
Extraction time: cpu 0.75 sec/real 91.56 sec = 1%. Speed 11.12 mB/s
All OK

[elit]

I am inflating Cyberpunk 2077 with cp2077 plugin. All *.archive tarred through groups in FA. I am at 32gb in tmp file right now and total of all archives is 60gb so I am still only half the size of original compressed data. This is after 6h10min. In FA gui it says speed is at 0.11mb/s. Is this supposed to be this slow? -t100p and 4 core 4690k 4.2ghz

PS(first ~27gb or so went much quicker than is now, but it is working and did not hanged)

[KaktoR]

@elit: No it's not supposed to be that long.

Overall input size: 59.64 GB
Overall output size: 125.20 GB
Overall conversion time: 01:23:28

[elit]

Exactly! But.. I doubt I am doing anything wrong, I mean:
-mcustom6/$gamelz=xtool:mcp2077+srp64+xlzma:lc8*.archive = $gamelz group.
So *.archive = FA -mxtool:mcp2077+srp64+xlzma:lc8
or xtool precomp -t100p -mcp2077 to be precise
[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t100p {options} - - <stdin> <stdout>
unpackcmd = _EC\xtool\xtool decode -t100p - - <stdin> <stdout>srp64 = srep64:m3f:c256:mem8g:m64k:b16m:a2
procdefault = srp64+xlzma:lc8
-mcustom6/$gamelz=xtool:mcp2077+procdefaultAnyone?(PS its latest 2012r2)
EDIT(restarted without t100p, but I don't think that's it: 28650
How you guys getting ~10mb/s is beyond me, are you tarring or doing individual files? As it may matter..)
EDIT2(speed down to 0.6mb/s, estimation ~22h+ and slowing down. Really, this ain't right but tool appear to be working correct not hanging! Can anyone try: latest xtool version with cp2077 plugin and on tarred archives through FA not individual files? I am sure there is something wrong in one of those variables.)
PS(I forgot to note that game is gog with 1.06 update, that means 9gb 1.05 was also applied, maybe it changed oodle structure? I will postpone compression until solution/cause is clear.)

[elit]

So I just quickly tried oo2rect. That one seem to be working ok! I am going to use it on whole game and will see. I also tried xtool with -mkraken instead cp2077 plugin and also on individual archives. Speed issue remains. It does seem to have problem no matter what, while oo2rect seems fine.

EDIT:
oo2rect is same slow, after ~1h about 20gb and 0.xxmb/s speed. At this point I am pretty sure problem is not xtool nor oo2rect or plugin, but a game. If this was not a case with original version 1.03, try to update to 1.06, I think they changed data enough to become problem. For reference, files of concern are:
basegame_3_nightcity.archive
basegame_3_nightcity_gi.archive

[elit]

Alright so I left it overnight. I can confirm it to work correctly but it took 11h:40min to compress, from that about ~10h due to xtool oodle. Archive tested ok and decompression was much quicker, compression is that slow though.

[KaktoR]

No Idea what's wrong, but for me it's working just fine.

I use GOG version 1.06, along with an Ryzen 5 2600 and 16GB RAM and latest xtool version posted a few days ago.

I can test the whole game folder if you wish, but I can tell you the results will be just fine and nothing wrong.

Xtool settings I use:
[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mcp2077 -c128mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,714,137,974 bytes. Ratio 285.42%
Compression time: cpu 9.34 sec/real 704.44 sec = 1%. Speed 11.30 mB/s
All OK

Extracted 1 file, 22,714,137,974 => 7,958,224,896 bytes. Ratio 285.42%
Extraction time: cpu 8.11 sec/real 333.60 sec = 2%. Speed 23.86 mB/s
All OK

[elit]

Hm, you have 6 core with hyperthreading. Maybe -t100p for you utilize all 12 virtual cores. You may try -t4 and also no --dbase to see if it affect speed that much. Also no need to test all archives only 2 files I mentioned above is enough. I wonder if --dbase or 12 virtual cores make for such difference. BTW thanks for the info above, it's interesting to know.

[Razor12911]

i5 4590, 4 Cores (4 Threads)

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mcp2077 -c128mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,919,265,672 bytes. Ratio 287.99%
Compression time: cpu 9.91 sec/real 1538.09 sec = 1%. Speed 5.17 mB/s
All OK

Tested 1 file, 22,919,265,672 => 7,958,224,896 bytes. Ratio 287.99%
Testing time: cpu 6.45 sec/real 528.38 sec = 1%. Speed 15.06 mB/s
All OK

My concern is the difference in ratio, --dbase should not have an impact on ratio just the speed. Some bug I guess...

[Masquerade]

Here's an interesting one.

ZZ1.dat - 175MB from Steel Division 2.


XTool 2020 (zlib)
Compressed 1 file, 183,623,680 => 183,661,889 bytes. Ratio 100.02%
Compression time: cpu 0.16 sec/real 26.08 sec = 1%. Speed 7.04 mB/s
All OK

Xtool v12 (zlib)
Compressed 1 file, 183,623,680 => 226,689,420 bytes. Ratio 123.45%
Compression time: cpu 0.27 sec/real 26.18 sec = 1%. Speed 7.01 mB/s
All OK
XTool v12 hangs inifintely when during decompression - the file just keeps getting bigger and bigger.

GFS Detects zlib streams, whereas Drop Scan does not:

https://i.imgur.com/YHG8hpl.png

https://i.imgur.com/6n0GrEA.png

https://anonfiles.com/FcY1Q060p9/ZZ_1_dat

[KaktoR]

Make headerless+force detection

PS: The anonfiles is shit. Always slow connection here :D (185kb/s)

[KaktoR]

@Masquerade
Use reflate codec

Compressed 1 file, 183,623,680 => 227,188,297 bytes. Ratio 123.72%
Compression time: cpu 0.09 sec/real 5.12 sec = 2%. Speed 35.87 mB/s
All OK

Extracted 1 file, 227,188,297 => 183,623,680 bytes. Ratio 123.72%
Extraction time: cpu 0.13 sec/real 1.07 sec = 12%. Speed 171.41 mB/s
All OK

[Razor12911]

When you use zlib method in Xtool v12 or older variants of xtool, they automatically used reflate when you placed the libraries each time zlib fails, the new xtool does not do any of this. The user is the one who needs to do this by themselves because what I have noticed is people putting reflate libraries near xtool and they never know when xtool uses them or when it doesn't. This way the user knows what the issues is when decompression fails each time when they try to figure out what the actual problem is.

I also did point out that the best method for precompressing anything that is zlib/deflate compressed, use -mzlib along with either reflate/preflate so you get the best results.

[Masquerade]

Razor12911,
I did use zlib+preflate, I simply forgot about reflate.

After using relate, everything works fine.

[Razor12911]

Update available

Changes

- updated library support
- updated command line parser
- included x86 build
- fixed depthing issues

Notes

I have added 32-bit build as requested but you'll have to use x86 libraries or find a matching pair of x86/x64 libraries. You can get these from github of a project under the releases section.
The versioning is brought back to the format of 1.0.0 instead of the build number as requested.

[Masquerade]

@Razor12911

Thanks for this new method... incredible!

Compressing 1 file, 17,530,888,054 bytes
Compressing TSCGame-WindowsNoEditor.pak
Compressed 1 file, 17,530,888,054 => 28,460,607,223 bytes. Ratio 162.35%
Compression time: cpu 18.42 sec/real 624.55 sec = 3%. Speed 28.07 mB/s
All OK
On the sample I sent to you yesterday.

[Cesar82]

@Razor12911, thanks for the 32-bit version:eek:, it will be very useful to include in CIU.
1) Also like to inform you that it is being detected as a false positive by the KIS anti-virus (I don't know if there is anything that can be done).
P.S: The 64-bit version has not been captured.
https://i.imgur.com/rdaSySe.png
2) You could share libraries compatible with the 32-bit version to be fully compatible and avoid errors due to incompatibilities.
3) I wonder if I have 3 games (collection) and each game uses XTool with different methods to compress.
Assuming that only the libraries needed for compression (nothing more than necessary) for each Data.bin file are together with XTool.exe at the time of compressing.
To perform the decompression, can I have other libraries that were not used to compress the Data.bin file together with XTool (Like: Have 1 library next to XTool when compressing and 5 when decompressing)?
Thanks!

[Razor12911]

1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/9f1779523fdc74df835c9d500ad6fc89fbcecfc0d6d51db0dd ca4cc87fff296d/detection
x64: https://www.virustotal.com/gui/file/546a063ab36874d59c4b298c183db26112a003be3369405609 8f6b8d224f7dfe/detection

the code is actually the same so I don't know... :D

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
28717

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.

[Cesar82]

1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/9f1779523fdc74df835c9d500ad6fc89fbcecfc0d6d51db0dd ca4cc87fff296d/detection
x64: https://www.virustotal.com/gui/file/546a063ab36874d59c4b298c183db26112a003be3369405609 8f6b8d224f7dfe/detection

the code is actually the same so I don't know... :D

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
28717

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.
1) For me it displays "Item not found" message in the virustotal links of your post.
Thank you for the informations.
The false positive does not happen when using Windows Defender. I don't understand why KIS is capturing him. (Just because it's x86 (same source code)... It's like COVID, it reaches the weakest organisms (x86)).

2) I know that few use x86 systems. But because CIU still works on x86 systems it is useful to have x86 compatibility.
Maybe in some time, by default, in the CIU, the "ArchitecturesInstallIn64BitMode=x64" and "ArchitecturesAllowed=x64" configs would limit the CIU to only 64-bit systems so only 64-bit versions of compressors would be needed.
But for now, if possible, I prefer to maintain compatibility.
I know that some plugins/libraries only exist 64-bit versions like the Kraken method that requires oo2core_#_win64.dll (I think there are no 64-bit versions of these libraries).

3) I see in your examples that zlibwapi.dll is always with XTool.
Is this library necessary to be together with XTool in methods other than zlib?
I ask why I want to keep the libraries that it is not mandatory to always be with XTool in a subfolder and depending on the method selected for compression before compressing, a copy of these libraries will be made to XTool. After compressing, these copied libraries will be deleted. In the installer decompressors there will be all libraries used with XTool to compress all Data # .bin that will be extracted by the installer. So I asked the previous question (already answered), if I could have other libraries that were not used in the compression method with XTool when performing the extraction.

Thanks for the great job.

[Razor12911]

1) Yeah a bit unfortunate
2) There are 32-bit versions just that the games usually never ship with them as they are unused and they are 64-bit anyways.
3) As you say "examples", it's just an example that just happen to require zlibwapi.dll if you don't use the method zlib you are not required to include the library

[Cesar82]

@Razor12911
Does XTool 2020 already support the "lz4" and "zstd" methods? Or do you have any limitations for these 2 methods?
If so, is this usage correct in arc.ini?
[External compressor:xtool_lz4]
header = 0
packcmd = xtool.exe precomp -mlz4 -c32mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

[External compressor:xtool_zstd]
header = 0
packcmd = xtool.exe precomp -mzstd -c32mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

@Razor12911, See your inbox.
I will send you a PM with some results using XTool with some samples that KaktoR sent me (Some tests did not inflate).

[Razor12911]

Update Available

Changes

- improved depthing
- updated library support
- fixed zstd codec issues
- removed fast memory

@Cesar82

-mzstd should work now, as for -mlz4. A plugin for that specific game or game engine should be made, the community can send in samples for games and I'll take a look at them if I can add support or not.

[Razor12911]

Update Available

Changes

- updated lz4 codec
- updated library support

[KaktoR]

Thanks for all these informations. Very usefull to us ;)

[Razor12911]

Update available

Changes

- fixed bug depthing (thanks dixen)

[Razor12911]

Update available

Changes

- updated oodle codec (fixed lzna bug)
- added custom method configuration

Notes

Custom method configuration is that xtool.ini file near the executable where you can put all the method which can't be passed directly via Freearc when using {option} feature in arc.ini.

xtool.ini
[CustomMethods]
borderlands3=-mzlib+ue4:m1:k115EE4F8C625C792F37A503308048E79726E 512F0BF8D2AD7C4C87BC5947CBA7 -d1
hitman3=-mlz4+hitman3 -d1
nfsheat=-mfrostbite3:nfsheat

if you want to compress borderlands for example, the command line via FA should look like this

arc.exe a -ep1 -r -ed -s; -w.\temp -mxtool:borderlands3 data.arc "pack\*"

xtool will check if the codec exist somewhere in the configuration then if it does it will convert it "-mborderlands3" to whatever is written in the configuration.

This feature was added to help Cesar's DiskSpan GUI project but it can have several uses. If methods become too many, users can add all the methods in xtool.ini so they know what they are for and use them by referring to them by their stored names.

[Cesar82]

@Razor12911, thanks for the new feature.
It will be very useful to include the methods of the plugins, but for the ue4 plugin it is necessary to send the key and parameter -m# because the decryption key can be used for any other game that is not listed.
There will be some changes to the "fcnd" plugin that KaktoR sent me, or it is finished (it was huge)?

[Razor12911]

You can always use SetIniString('CustomMethod','ue4customkey','-mzlib+ue4:m1:some key of a game that is not listed','xtool.ini');, basically you write the key the user has provided and each time use it as -mue4customkey.

as for fcnd, a few things need to be fixed.

[Cesar82]

You can always use SetIniString('CustomMethod','ue4customkey','-mzlib+ue4:m1:some key of a game that is not listed','xtool.ini');, basically you write the key the user has provided and each time use it as -mue4customkey.

as for fcnd, a few things need to be fixed.

Yes I thought about it (setinistring), but I would have to completely separate the command line for the unreal plugin.
See in the VIDEO (https://mega.nz/file/kAJTkSrQ#EDuWo2f_APP9Ox84D6RE5sLg3r2hP7tGYf7lR6lKT vY) how it works on DiskSpan_GUI. It works the same way for ue4dt.

The Arc.ini file was fixed with Zlib.
[External compressor:xtool_ue4]
header = 0
packcmd = "PRE\XTool_2020\Win64\XTool.exe" precomp -mzlib+ue4{:option} -c32mb -t100p -d1 --dbase - - <stdin> <stdout>
It may be possible to check the method (s) prior to ue4dt and if it is an XTool 2020 method it could be cut (the previous ones together) and sent to XTool.ini and replaced with the name of the key (as an example ue4method=).

I'll think of something later, but this feature of XTool.ini is very useful.
Should I always use the name XTool.ini or if I rename XTool.exe to XTool_x64.exe should I also rename the INI?

[go2top]

Great work that makes me thrill enough to keep up the good work. If you want to make your brand popular just take service from the Go2Top Panel.

[Razor12911]

Update available

Changes

- updated oodle codec (fixed more lzna bugs)

[Masquerade]

Is there a way to cap memory usage by xtool?

I ask because I have wasted about 5 hours with Project Cars 3 Kraken codec, where it would get to nearly finished precompressing, then shoot up in memory usage completely crashing my PC needing force restart from the reset button on my case. (Bear in mind this is with xtool capped at only 6 threads of CPU, so memory usage isn't too bad anyway). Oodlerec is terribly slow.

Same thing happens with American/Euro Truck Simulator, in those cases I have no choice to use a chunk size of 8mb otherwise my PC just crashes.

Thanks!

Edit: I get past this just by swapping STDIO for $$arcdatafile$$.tmp $$arcpackedfile$$.tmp - seems good!

[Razor12911]

Update available

Changes

- updated library structure

[Razor12911]

Update available

Changes

- fixed command line parser bug
- updated library support

[github]

Hi thanks for the great work.

With xtool0.12 I have


[External compressor:xzlib,xZLib]
header = 0
packcmd = "pre\xt12\xtool" e:precomp:t75p,c128m:zlib - - <stdin> <stdout>
unpackcmd= pre\xt12\xtool d:precomp:t4 - - <stdin> <stdout>


On a testing file set mostly zip and 7z files, it could precomp it well

;xzlib:
;Compressed 139 files, 822,152,142 => 1,096,061,919 bytes. Ratio 133.32%
;Compression time: cpu 0.81 sec/real 31.35 sec = 3%. Speed 26.23 mB/s


Then I could add srep+FL2 to compress it with a good ratio.

With the recent release xtool 2020 from xtool_0.3.8.7z,

with a setting of


[External compressor:xzlib2,xZLib2]
header = 0
packcmd = "pre\XTool_2020\XTool" precomp:zlib:c128m,t75p - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>


It will not do the similar precomp process, output is


;xzlib2
;Compressed 139 files, 822,152,142 => 822,151,338 bytes. Ratio 100.00%
;Compression time: cpu 0.53 sec/real 2.23 sec = 24%. Speed 369.24 mB/s


Do I have to adjust/add the parameter to have the similar results?

Edit:

I guess the default values for zlib might have changed. And on some aspects the new version is performing much more like a normal average setting.


For a set of .dat files 800MB, tried to compressing with arc

srep+FL2 compressed it to 414300KB, takes 32 seconds.

xtool_0.12 e:precomp:t75p,c128m:zlib+srep+FL2 compressed to 412000KB takes a big 55 seconds. the zlib codec here is working very hard on something and found a little bit to more to compress further. But it added 23 seconds up on (srep+FL2)

xtool_0.12 e:precomp:t75p,c128m:zstd+srep+FL2 got 414000KB takes 36 seconds, it adds 4 seconds up on (srep+FL2)

xtool_0.3.8 e:precomp:t75p,c128m:zlib+srep+FL2 got to 414000KB takes 33 seconds, it adds 1 seconds up on (srep+FL2)

xtool_0.3.8 e:precomp:t75p,c128m:zstd+srep+FL2 got to 414000KB takes 33 seconds , it adds 1 seconds up on (srep+FL2)

[Razor12911]

Update available

Changes

- fixed future stream bug

@github

The command line has changed, check the example to see how it's used.

Furthermore, the old xtool used reflate without the knowledge of the user. The new one doesn't so if you have a scenario where the old xtool gives better results, then you may need to combine zlib with reflate or preflate.

[DiCaPrIo]

small test on WDL

[github]

Thank you. So with xtool 2020 I should use zlib+reflate -d or zlib+preflate -d
on command line with this it does detect many streams


.\xtool.exe precomp -mzlib+reflate -d3 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
14992 streams
.\xtool.exe precomp -mzlib+preflate -d5 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
2603 streams


how to I pass the -d level in the arc.ini?
does xtool0.12/2020 support the writing of {options} in arc.ini?

I tried


packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1:d3 - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1,d3 - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1 {options} - - <stdin> <stdout>

packcmd = "pre\XTool_2020\XTool" precomp:mzlib+reflate:c128m,t100p-1 -d3 - - <stdin> <stdout>


Those doesn't make any difference to the one without d3 option

[Razor12911]

are you precompressing something that requires depth level to be set to 3 or higher? as for choosing between reflate or preflate, you need to know how each works as they have their advantages and disadvantages.

.\xtool.exe precomp -mzlib+reflate -d3 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
14992 streams
.\xtool.exe precomp -mzlib+preflate -d5 -c128mb -t100p-1 D:\xtool2020\test.zip output.bin
2603 streams

which should explain their differences seen here. What I would point out as the main difference, reflate supports streams with no tail while preflate doesn't so if a special deflate encoder was used or if a stream had been cut off, preflate will be unable to process such streams but reflate is able to handle these.

as for {option} via freearc, I'd first say you need to understand the command line syntax of xtool. The old xtool and new xtool use different syntax.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib -c32mb -t100p - - <stdin> <stdout>

Ideally this would become

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp {option} - - <stdin> <stdout>

then used as -mxtool:mzlib:c32mb:t100p from freearc.

[github]

Thank you for the example.

I have it setup up like this


;zlib zstd,oodle,reflate,preflate
;calling example: xtool:mzlib or xtool:mzlib:c128m:t100p-1 not xtool:mzlib+zstd
[External compressor:xtool]
header = 0
default = -c128m -t100p-1
packcmd = "pre\XTool_2020\XTool" precomp {options} - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>



; calling example zlib:c128m:t100p-1
[External compressor:zlib,zstd,oodle,reflate,preflate]
header = 0
default = -c128m -t100p-1
packcmd = "pre\XTool_2020\XTool" precomp -m{compressor} {options} - - <stdin> <stdout>
unpackcmd= "pre\XTool_2020\XTool" decode:t4 - - <stdin> <stdout>



However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

[Razor12911]

However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.
This true.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

I wonder though, A depth of 3-5. I asked before if there is such data you have on hand because this is basically the same as having data, compress it using zlib, the output, compress that using zlib, do this at least 5 times. That's what depth is.

A depth of 1-2 is usually enough on everyday files such as pictures and documents as these are usually shipped in a zip file, that zip file is compressed using deflate and if it contains pdfs or png images then, that's when you need to use depth otherwise, don't touch it as you'll be increasing precompression time unnecessarily and have no gains in ratio.

[Masquerade]

Hello Razor, is there anything that can be done to cap the memory usage of xtool at a certain amount?

I am aware of a cmem variable in arc.ini packcmd (only small knowledge) however I was wondering if there's any other ways about this.

I find on larger workloads/workloads with high amounts of streams, xtool fills up my RAM and completely crashes my PC.

I am working with 16GB memory. Is there anything that can be done without changing chunk size?

Currently I'm capping threads and using $$arc$$ instead of stdio.

[Razor12911]

-lm

[Masquerade]

:D:D thank you for implementing such a feature, I thought it got removed when the project was started (I remember it being in 2019 version).

Thanks again.

Compressed 1 file, 12,630,693,406 => 54,088,313,477 bytes. Ratio 428.23%
Compression time: cpu 66.05 sec/real 1078.31 sec = 6%. Speed 11.71 mB/s
All OK

[Snake288]

Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method

[Razor12911]

Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method

you can't cap the memory usage of the program, you can only reduce it. When you are compressing, usually each thread gets its own chunk. Using -lm parameter makes all thread use 1 chunk. As for memory usage, that's highly dependent on the user and the data they are precompressing, if you set very high chunk sizes, be prepared for high memory usage. When decompressing, xtool tries to not use more than 512 MB ram automatically unless if there are large streams or if the recompressing library requires more memory.

[doofoo24]

did quick test on mad max with/out --dedup=xtool.bin to se mem usage...
with --dedup size to 55gb
without --dedup size 57.1gb
but memory dec the same around 6gb.
*note only test with mzlib+srep

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib -c128mb -t12 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>


[External compressor:srep]
header = 0
packcmd = srep.exe -m3f -d1g -a2 $$arcdatafile$$.tmp - <stdout>

[FitGirl]

Mad Max is a bad example for dedup testing. They use really huge deflate chunks with level 1 compression, so the amount of dupe chunks is not very big, unlike the duplicated data in those chunks when decompressed, that's why srep is good solution here.

[doofoo24]

so game like (just cause 3 &4, ac syndicate...) using dedup xtool will give similar results as mad max :(
so i guess dedup xtool most useful on small chunks :confused:

[dixen]

For future repacks Borderlands 3))

pakchunk0-WindowsNoEditor_0_P.pak

packcmd = xtool.exe precomp -mzlib+preflate -c128mb -t100p-1 - - <stdin> <stdout>

Compressed 1 file, 415,114,853 => 539,243,092 bytes. Ratio 129.90%
Compression time: cpu 0.41 sec/real 8.27 sec = 5%. Speed 50.17 mB/s
All OK

packcmd = xtool.exe precomp -mzlib+preflate -d3 -c128mb -t100p-1 - - <stdin> <stdout>

Compressed 1 file, 415,114,853 => 1,540,405,512 bytes. Ratio 371.08%
Compression time: cpu 0.42 sec/real 106.39 sec = 0%. Speed 3.90 mB/s
All OK

Final compress with srep+lolz

With -d3 - 61 mb
Without -d3 - 270 mb



NOTE.
Before precompress *.pak is decrypted with ue4dt
Used XTool v0.3.9

[Hexagon123]

For some reason, they gave me an decompression failed corrupted archive error by decrypting the pak file.

unarc.dll returned -11

I put a config on Bunti's Windows Phone Bug Free Installer arc.ini file.

Is there anything wrong?

Solved by adding a new resource.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+ue4:m1:kC40250B917281AC9874D93C53E429255177E A6E038A117054A1F1C5DA92AB26A -c128mb -t75p -d3 --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t75p - - <stdin> <stdout>

[L0v3craft]

Hi. Someone tested xtool 0.3.9 with the command "-mlz4" with some game and it worked? I'm testing it with several games made with unity engine and it doesn't work. Tested also with all the downloadable lz4 dll (i have downloaded from github versions 1.7.3, 1.7.4, 1.7.4-2, 1.8.0, 1.8.1.2, 1.8.2, 1.8.3, 1.9.3). Looks like xtool doesn't try either to precompress the files, because it is very fast, maybe it is ignoring that command "-mlz4"? Is that command a placeholder or what?

[Hexagon123]

Hi. Someone tested xtool 0.3.9 with the command "-mlz4" with some game and it worked? I'm testing it with several games made with unity engine and it doesn't work. Tested also with all the downloadable lz4 dll (i have downloaded from github versions 1.7.3, 1.7.4, 1.7.4-2, 1.8.0, 1.8.1.2, 1.8.2, 1.8.3, 1.9.3). Looks like xtool doesn't try either to precompress the files, because it is very fast, maybe it is ignoring that command "-mlz4"? Is that command a placeholder or what?

Maybe, try using the old XTool?

Some use LZMA, but some use LZ4?

Seems like razor was planning on it?

[PsYcHo_RaGE]

@Moderators
Don't you guys think that this thread should be pinned?

[Razor12911]

There'll be more sticky threads than normal threads per page )

@Moderator
unstick these threads
https://fileforums.com/showthread.php?t=94605
https://fileforums.com/showthread.php?t=96630

Thanks

[Grumpy]

There'll be more sticky threads than normal threads per page )

@Moderator
unstick these threads
https://fileforums.com/showthread.php?t=94605
https://fileforums.com/showthread.php?t=96630

Thanks

I unstuck those 2 and made this thread a Sticky ;)

[Gehrman]

[External compressor:xtool]
header = 0
packcmd = xtool_0.3.9\xtool.exe precomp -mkraken -c32mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool_0.3.9\xtool.exe decode -t100p - - <stdin> <stdout>

Tested games
Gears Tactics
Assassin's Creed Valhalla:D

[Masquerade]

Why are you using an Oodle kraken precompressor when Valhalla is compressed using Oodle Mermaid? :confused:

[Gehrman]

Why are you using an Oodle kraken precompressor when Valhalla is compressed using Oodle Mermaid? :confused:

DataPC_extra_patch_01.forge 529MB
Creating archive: data.arc using xtool
Compressed 1 file, 555,122,688 => 1,072,196,303 bytes. Ratio 193.15%
Compression time: cpu 0.64 sec/real 64.80 sec = 1%. Speed 8.57 mB/s
All OK

Testing time: cpu 0.75 sec/real 44.12 sec = 2%. Speed 12.58 mB/s
All OK
____________________________

Creating archive: data.arc using oo2recm
Compressed 1 file, 555,122,688 => 1,173,235,688 bytes. Ratio 211.35%
Compression time: cpu 0.58 sec/real 229.01 sec = 0%. Speed 2.42 mB/s
All OK

Testing time: cpu 0.52 sec/real 52.80 sec = 1%. Speed 10.51 mB/s
All OK
____________________________

Creating archive: data.arc using xtool+srep:m3f:l512+4x4:t10:i0:lzma:64mb:normal:32
Compressed 1 file, 555,122,688 => 413,916,053 bytes. Ratio 74.56%
Compression time: cpu 536.95 sec/real 125.05 sec = 429%. Speed 4.44 mB/s
All OK

Testing time: cpu 38.48 sec/real 49.68 sec = 77%. Speed 11.17 mB/s
All OK
____________________________
Creating archive: data.arc using oo2recm+srep:m3f:l512+4x4:t10:i0:lzma:64mb:normal: 32
Compressed 1 file, 555,122,688 => 403,297,897 bytes. Ratio 72.65%
Compression time: cpu 559.47 sec/real 295.72 sec = 189%. Speed 1.88 mB/s
All OK

Testing time: cpu 37.36 sec/real 58.93 sec = 63%. Speed 9.42 mB/s
All OK

[kj911]

packcmd = xtool.exe precomp -mzlib+preflate -d3 -c128mb -t100p-1 - - <stdin> <stdout>
...
With -d3 - 61 mb
Without -d3 - 270 mb
...
Used XTool v0.3.9

Old XTool 0.1.2 x86 version capable the preflate compression schema? (This -d3 switch preflate or zlib command?) Required external EXE/DLL files?

[ozolt]

For some reason xtool does not precompress /detect zlib streams in *.map files from halo2 folder and *.s3pak from halo1 folder (Halo: The Master Chief Collection).
Examples in attachments.
And xtool 0.12 detects them and inflates them just fine!
Tried all possible codec combos in xtool 0.3.9, -mzlib+reflate, -mzlib+preflate, -mzlib etc... no working. -mreflate works on other *.map files from halo3, halo4 perfectly...But not on halo2 (*.map files) and halo1 (*.s3pak files).
Since xtool 0.12 uses zlib+reflate xtool 2020 should detect this also...maybe there could be some *.ini plugin for 2020 to detect this streams?

Thanks,
regards.

[pratikpatel8982]

For some reason xtool does not precompress /detect zlib streams in *.map files from halo2 folder and *.s3pak from halo1 folder (Halo: The Master Chief Collection).
Examples in attachments.
And xtool 0.12 detects them and inflates them just fine!
Tried all possible codec combos in xtool 0.3.9, -mzlib+reflate, -mzlib+preflate, -mzlib etc... no working. -mreflate works on other *.map files from halo3, halo4 perfectly...But not on halo2 (*.map files) and halo1 (*.s3pak files).
Since xtool 0.12 uses zlib+reflate xtool 2020 should detect this also...maybe there could be some *.ini plugin for 2020 to detect this streams?

Thanks,
regards.
I personally use pzlib3:m1 for zlib streams, it has worked well for me so far.
infile: 14.1GB
srep:m3f+pzlib3:m1+lolz
Outfile:1.80GB

Also, first tempfile was 14.1GB (srep)
Second tempfile was 2.11GB (pzlib3)
Final file was 1.80GB (lolz ultra)

I ALSO TRIED VERSION OF XTOOL THAT COMES WITH ULTRA ARC BUT IT DIDN'T ACHIEVE GOOD RESULTS.

[Cesar82]

@Razor12911, I had an idea for XTool.
I don't know if that would be possible, but since we can't pass the "+" character in a parameter on the FreeArc command line, I thought it might be possible to make XTool decode multiple method parameters separately, allowing you to send multiple commands in a bat file...

Currently the parameter for the XTool method is -m.
Currently it is possible to use this way:
Arc.ini
[External compressor:xtool]
header = 0
default = -c64mb -t100p
packcmd = "PRE\XTool_2020\XTool.exe" precomp {options} --dbase - - <stdin> <stdout>

pack.bat
arc.exe a -ep1 -r -ed -s; -w.\temp -mxtool:mzlib data.arc "pack\*"


Suggestion is to use multiple parameters (XTool concatenated with "+" sign the parametters -m1, -m2, -m3, etc if it exists, otherwise it would use -m current).
This change in XTool would allow you to use combinations (without parameters)

Arc.ini
[External compressor:xtool]
header = 0
default = -c64mb -t100p
packcmd = "PRE\XTool_2020\XTool.exe" precomp {options} --dbase - - <stdin> <stdout>

pack.bat
arc.exe a -ep1 -r -ed -s; -w.\temp -mxtool:m1zlib:m2reflate data.arc "pack\*"

The result would be as if using mzlib+reflate


To make it possible to send parameters, "," could be used in the parameters -m1, -m2, etc. and XTool replaces ":" internally when concatenating the methods.

Borderlands 3 method if usage example if I applied my idea to XTool
pack.bat

arc.exe a -ep1 -r -ed -s; -w.\temp -mxtool:m1zlib:m2ue4,m1,k0x115EE4F8C625C792F37A5033 08048E79726E512F0BF8D2AD7C4C87BC5947CBA7 data.arc "pack\*"

This is just an idea.
I don't know if it would be good for XTool or if for some reason it would hinder performance so think about this idea.

This would be very useful to use with DiskSpan_GUI as I cannot send combined methods currently. And using XTool.ini is complicated because DiskSpan_GUI is fully configurable in the INI file without internal setinistring processing with sections and fixed keys to insert these values ​​in XTool.ini (You would have to read the DiskSpan_GUI configuration INI file to know where it would be inserted the value).

[Razor12911]

Update available

Changes

- added depth info functions
- added support for oodle 2.9.0+ functions
- fixed data patching bug
- updated oodle codec
- updated command line parser

Changes between 0.3.9 to 0.3.11

ES_R11 (0.3.11)
- fixed x86 build bugs
- fixed config multi-threading bug
- fixed resource management bug
- fixed deduplication bug

ES_R10 (0.3.10)
- minor bug fixes
- added diff tolerance parameter (--diff=)
- fixed plugin database bug
- fixed lz4 codec bug
- updated oodle codec
- updated library structure
- added resource management
- added direct use encryption codecs
- added embedded deduplication feature (--dedup) [makes temps during encoding]

@Cesar82

I updated the command line parser

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

If you use "-mxtool:zlib:ue4,m1,k0x115E..." via Freearc, this is passed to xtool as "-mzlib+ue4:m1:k0x115E..." which should allow you to use it in your project.

[Razor12911]

@Everyone

The oodle precompressor in xtool is now more aggressive (means slower), I'll add aggression parameter to control how much time it should spend finding more and more streams however this only applies to kraken, mermaid, selkie and hydra. Leviathan is still problematic and you'll have to rely on plugins for data compressed using this codec.

Here are a few benchmarks that show what I am talking about:

0.99 GB (1,066,131,456 bytes) > 1.52 GB (1,638,079,260 bytes) [took 2 minutes, 9 seconds] (xtool 0.3.9)
0.99 GB (1,066,131,456 bytes) > 1.84 GB (1,977,644,853 bytes) [took 4 minutes, 56 seconds] (xtool 0.3.12)
0.99 GB (1,066,131,456 bytes) > 1.53 GB (1,649,308,007 bytes) [took 8 minutes, 52 seconds] (oo2reck)

As you can see, version 0.3.12 is now slower but it found more streams than 0.3.9 or the oodle precompressor side project.

This is a sample from Cyberpunk 2077 and since the game does have a plugin these are the results if you had use the plugin

0.99 GB (1,066,131,456 bytes) > 2.26 GB (2,432,780,012 bytes) [took 41 seconds] (plugin)

So what do these results mean? Well the newer version of xtool traded speed for more ratio in an attempt to beat the side project while still being faster. Also if a plugin for a specific game is created, it shows that the plugins will constantly be superior to the universal precompressor as it's not only faster but produces better results. :)

In conclusion, try to run tests with other games to see if the new xtool produces better results than oo2rec and if a plugin for a game exists then it's best to just use it.

[:( Sad8669]

I am now even more fired up for FIFA.

Thanks for these updates.

[Razor12911]

For FIFA and all these other frostbite games, 0.3.13 update is needed

[:( Sad8669]

Crash Bandicoot 4 Comparison - For zlib streams

0.3.12
Compressed 1 file, 19,665,315 => 91,532,563 bytes. Ratio 465.45%
Compression time: cpu 0.05 sec/real 2.77 sec = 2%. Speed 7.11 mB/s
All OK
0.3.9
Compressed 1 file, 19,665,315 => 98,920,152 bytes. Ratio 503.02%
Compression time: cpu 0.06 sec/real 2.80 sec = 2%. Speed 7.03 mB/s
All OK


Those FIFA's after 19 are such a pain...

[Razor12911]

Crash Bandicoot 4 Comparison - For zlib streams

0.3.12
Compressed 1 file, 19,665,315 => 91,532,563 bytes. Ratio 465.45%
Compression time: cpu 0.05 sec/real 2.77 sec = 2%. Speed 7.11 mB/s
All OK
0.3.9
Compressed 1 file, 19,665,315 => 98,920,152 bytes. Ratio 503.02%
Compression time: cpu 0.06 sec/real 2.80 sec = 2%. Speed 7.03 mB/s
All OK


Those FIFA's after 19 are such a pain...

must be deduplication, check command line

FIFA after 19 isn't a problem, I don't have samples for these games so I am limited by the little samples that I have.

[Cesar82]

@Cesar82
I updated the command line parser

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

If you use "-mxtool:zlib+ue4,m1,k0x115E..." via Freearc, this is passed to xtool as "-mxtool:zlib+ue4:m1:k0x115E..." which should allow you to use it in your project.
@Razor12911, I don't know if another method besides ue4 has parameters. But supposing yes, how would I use a combination of 2 methods in which each one contains parameters?
Assuming that zlib had a parameter -m3 for example (just to specify the debt):

For example, assuming that zlib had a parameter -m3 (just to specify the debt) would it look like this?
"-mxtool:zlib,m3+ue4,m1,k0x115E..."

Should the arc.ini file be used in this way only { -moption}?
to use zlib+ue4 (Borderlands 3) it would not be necessary to send the -d1 parameter to XTool.
How can I pass or not the -d1 if I use the command like the arc.ini information reported above?

[Razor12911]

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

-mxtool:zlib,m3:ue4,m1,k0x115E...

If you plan to add -d1 option via command line, you then have to modify the arc.ini to something like

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -option} -c32mb -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

-mxtool:d1:mzlib,m3:mue4,m1,k0x115E

which equates to -d1 -mzlib,m3 -mue4,m1,k0x115E
then -d1 -mzlib+ue4:m1:k0x115E when it reaches xtool

[:( Sad8669]

Might hook you up with some samples, once i get to testing it. If you would like.

Yes it was the dedup.

Without dedup :
Compressed 1 file, 19,665,315 => 98,946,941 bytes. Ratio 503.15%
Compression time: cpu 0.00 sec/real 4.12 sec = 0%. Speed 4.77 mB/s
All OK

[KaktoR]

Bethesda plugin does not work with xtool 0.3.12

0.3.9
Compressed 1 file, 373,427,426 => 532,225,912 bytes. Ratio 142.52%
Compression time: cpu 0.25 sec/real 4.74 sec = 5%. Speed 78.73 mB/s
All OK

0.3.12
Compressed 1 file, 373,427,426 => 373,427,610 bytes. Ratio 100.00%
Compression time: cpu 0.41 sec/real 2.22 sec = 18%. Speed 168.52 mB/s
All OK

Yes I have replaced bsa.dll with the new one.

[21h2]

XTool is program made specifically repackaging games by providing a full suite of useful features such as data precompression, archiving, encryption and etc.

With that being said, nothing restricts it from being used on everyday files such as documents, pictures and media but with few limitations.

Read the documentation to find out how it works and how to use it.

Link (https://mega.nz/folder/03BkjKQA#WpTxyngxucC-ToBRK9xbkw) for older releases.

i still dont understand how i can zip thinks with it

[Razor12911]

Update available

Changes

- updated lz4 codec
- updated library structure
- updated depth info functions
- updated depth feature

[kast1450]

XTool LZ4:
8:20:52 PM - Overall input size: 6.54 GB
8:20:52 PM - Overall output size: 3.91 GB
8:20:52 PM - Overall conversion time: 00:00:52

XTool 2020 LZ4:
8:22:33 PM - Overall input size: 6.54 GB
8:22:33 PM - Overall output size: 6.54 GB
8:22:33 PM - Overall conversion time: 00:01:02

Is this normal or a bug? I'm using the same settings with the same dataset yet one LZ4 gives good compression and the other gives no compression.

[ScOOt3r]

If i wanted to use Oodle with xtool
would this be correct with the latest verison of xtool

[External compressor:xtool]
header = 0
packcmd = xtool\xtool.exe precomp -moodle -c256mb -t22 --dbase --dedup - - <stdin> <stdout>

cause it doesnt work for me... get a disc full error

thanks
ScOOt3r

[Razor12911]

XTool LZ4:
8:20:52 PM - Overall input size: 6.54 GB
8:20:52 PM - Overall output size: 3.91 GB
8:20:52 PM - Overall conversion time: 00:00:52

XTool 2020 LZ4:
8:22:33 PM - Overall input size: 6.54 GB
8:22:33 PM - Overall output size: 6.54 GB
8:22:33 PM - Overall conversion time: 00:01:02

Is this normal or a bug? I'm using the same settings with the same dataset yet one LZ4 gives good compression and the other gives no compression.
xtool hasn't had a working lz4 codec.

If i wanted to use Oodle with xtool
would this be correct with the latest verison of xtool

[External compressor:xtool]
header = 0
packcmd = xtool\xtool.exe precomp -moodle -c256mb -t22 --dbase --dedup - - <stdin> <stdout>

cause it doesnt work for me... get a disc full error

thanks
ScOOt3r

there is no codec with the name oodle however there is kraken, mermaid and selkie.

Edit: the parameter you used will require about 6GB ram if no streams are found and it will require even more if they are found

(256mb x 22 threads) + 256mb = 5,888MB ram usage and if the average inflation ratio is 200%, you're looking at 12GB+ ram usage by xtool. I'm letting you know before hand that the parameter you have input will make xtool use a lot of memory.

[ScOOt3r]

Thank you Razor12911, thanks for the breakdown

I will edit my arc.ini file and correct that.

cheers
ScOOt3r

[Razor12911]

Update available

Changes

- minor bug fixes

Changes between 0.3.13 to 0.3.15

ES_R15 (0.3.15)
- converted library support to unicode (don't know why I used ansi in the first place)
- added library support functions
- added rc4 encryption support

ES_R14 (0.3.14)
- fixed library support bug
- updated library structure

Hotfix uploaded

[L0v3craft]

Update available

Changes

- added rc4 encryption support

Hi. Which game is using rc4 encryption? So we can do some test. Thanks.

[Razor12911]

Update available

Changes

- fixed multi-threading bug

Hi. Which game is using rc4 encryption? So we can do some test. Thanks.

There is madness plugin I was busy with earlier to add support to games that use the Madness game engine. (Project Cars series/Automobilista 2)

I only have a few samples of Automobilista 2 and it seems to work there, but I was more interested in the first project cars game.

[doofoo24]

@Razor12911

any benefit from adding embedded deduplication feature (--dedup) [makes temps during encoding] ???
game like gta v it unpack to xtool.vm then just copy to arc $$$ while using all ram and ssd write cycle...

so is it better to disable --dedup ???

[Razor12911]

Update available

Changes

- fixed depth bug
- fixed library plugin bugs

@doofoo24
I'll look into the issue but you can use xtool without deduplication, someone has already made comments regarding xtool's memory hungry behaviour when using the feature as this is not supposed to happen.

[doofoo24]

@Razor12911
does xtool 0.3.18 work with mass effect legendary ?
only xtool in New folder (7) work "packcmd = xtool.exe precomp -mue3:m2 -t100p - - <stdin> <stdout>" ...

[Razor12911]

Yes it works but you need the unreal plugin and oodle library from the game.

[dixen]

@Razor12911
does xtool 0.3.18 work with mass effect legendary ?
only xtool in New folder (7) work "packcmd = xtool.exe precomp -mue3:m2 -t100p - - <stdin> <stdout>" ...

ME2 Textures7.tfc
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 519,052,402 => 1,490,583,040 bytes. Ratio 287.17%
Compression time: cpu 0.56 sec/real 16.36 sec = 3%. Speed 31.72 mB/s
All OK

Tested 1 file, 1,490,583,040 => 519,052,402 bytes. Ratio 287.17%
Testing time: cpu 0.47 sec/real 13.01 sec = 4%. Speed 39.90 mB/s
All OK

XTool v0.3.18 + UE-Plugin R5

[doofoo24]

yea my mistake i used unreal.dll from "New folder (7)" with xtool 0.3.18 having the same size as unreal.dll from R5 i thought both the same ...:o

so i compress all of mass effect 1+2+3 super fast during install on r9 5950x... around 9min install 71gb without bik...

xtool set to 100p but most of the time around 60% during install unlike during compress xtool was using all 32 thread so i guess there is bottelneck lolz or srep ?

to test it i need to use only xtool to see if it can use 32 thread in decode...

[infovs]

Is there any way you can pass ue3:m2 command line parameters to arc.exe with "standard" arc.ini provided with xtool?

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -t100p --dbase --dedup - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

Should be -mxtool:ue3:m2 with arc.exe in .bat, but it is not working...
When I modify arc.ini packcmd to xtool.exe precomp -mue3:m2 .... it is working ok, but with { -moption} it fails.

[Razor12911]

@infovs

-mxtool:ue3,m2

[Razor12911]

Update available

Changes

- updated lzo codec

[Masquerade]

Is there anyway to analyse a file to see which version of LZ4 it was compressed with?

I have a few unity asset bundles, 506MB worth. When decompressed using Asset Bundle Extractor, they decompress to 2.78GB.

If I try to throw these into XTool, it will make an archive of 693MB (~136% ratio). I get the same whether I use the 2017 lz4 dll, 2018 (v1.8.1.2 / v1.8.2) lz4 dll or the latest dll from Nov 2020.

https://1fichier.com/?hcsaopws7sfrktfka4z2

Here's the sample

[Masquerade]

Here's some more info on that unity sample I shared above:

Compressing 4_trialislands_scenes_4d_bunker.unity.bundle
Compressing 42,742,931 bytes with "xtool.exe" precomp -munity -c512mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
XTool is created by Razor12911

Streams: 1509/8732732
Time: 00:00:30 (00:06:28)
Memory: 317 MB (317 MB)
Memory: 128 MB (128 MB)
8%
Errorlevel=0

[Razor12911]

Update available

Changes

- fixed library support bug
- x86 build discontinued (has bugs from nowhere)

@Masquerade

Use an even older library for Unity games

https://github.com/lz4/lz4/releases/tag/v1.8.0

Also use v0.3.20+

And there is no way you can tell what version of lz4 was used, that's why I made xtool to allow users to change them and go with the one that they think is the best.

Results from the samples you sent:
Tested 6 files, 2,793,950,894 => 531,410,227 bytes. Ratio 525.76%
Testing time: cpu 0.47 sec/real 92.79 sec = 1%. Speed 5.73 mB/s

[Masquerade]

Results from the samples you sent:
Tested 6 files, 2,793,950,894 => 531,410,227 bytes. Ratio 525.76%
Testing time: cpu 0.47 sec/real 92.79 sec = 1%. Speed 5.73 mB/s

Thanks for the update. I can only get the results you are showing if I run xtool in non-solid mode however, which doesn't really work when trying to chain it with srep and lolz.

[Masquerade]

What's the method to use for lzo precompression?

I have xtool + lzo2.dll, with packcmd:

[External compressor: lzo]
header = 0
packcmd = xrool precomp -mlzo -c512mb -t16 - - <stdin> <stdout>

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressing 1 file, 51,286,661 bytes. Processed 0%
ERROR: write error (disk full?) in compression algorithm lzo

I have also tried using the xtool example with same result.

Test file: https://1fichier.com/?1lbx3d68cud09anxs39v

It contains zlib and lzo streams - from Bioshock Infinite (unreal engine 3).

Using uelr:

Compressed 1 file, 51,286,661 => 100,636,058 bytes. Ratio 196.22%
Compression time: cpu 0.08 sec/real 6.75 sec = 1%. Speed 7.59 mB/s
All OK

uelr+xtool for zlib:

Compressed 1 file, 51,286,661 => 153,379,576 bytes. Ratio 299.06%
Compression time: cpu 0.08 sec/real 12.26 sec = 1%. Speed 4.18 mB/s
All OK

[dixen]

What's the method to use for lzo precompression?

I have xtool + lzo2.dll, with packcmd:

[External compressor: lzo]
header = 0
packcmd = xrool precomp -mlzo -c512mb -t16 - - <stdin> <stdout>

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressing 1 file, 51,286,661 bytes. Processed 0%
ERROR: write error (disk full?) in compression algorithm lzo

I have also tried using the xtool example with same result.

Test file: https://1fichier.com/?1lbx3d68cud09anxs39v

It contains zlib and lzo streams - from Bioshock Infinite (unreal engine 3).

Using uelr:

Compressed 1 file, 51,286,661 => 100,636,058 bytes. Ratio 196.22%
Compression time: cpu 0.08 sec/real 6.75 sec = 1%. Speed 7.59 mB/s
All OK

uelr+xtool for zlib:

Compressed 1 file, 51,286,661 => 153,379,576 bytes. Ratio 299.06%
Compression time: cpu 0.08 sec/real 12.26 sec = 1%. Speed 4.18 mB/s
All OK

https://fileforums.com/showpost.php?p=465943&postcount=73

[Masquerade]

https://fileforums.com/showpost.php?p=465943&postcount=73

Yes, I also used uelr previously when I compressed this game, but I want to test xtool lzo.

[ffmla]

^^
Try lzo with Far cry 3.

[External compressor: lzo]
header = 0
packcmd = xrool precomp -mlzo -c512mb -t16 - - <stdin> <stdout>
spelling...

----------------------
old xtool LZO usage

[External compressor:deflate,reflate,unity,lzo1c,lzo1x,lzo2a]
packcmd = xtool.exe precomp:{compressor}:c16mb,lm,t90p - - <stdin> <stdout>
unpackcmd = xtool.exe decode:t100p - - <stdin> <stdout>

arc.exe a -ep1 -r -ed -s; -w.\temp -mlzo1x data.arc "X:\datatobepacked\*"

[Masquerade]

spelling...
This was an error when I typed the post and not from my arc.ini.

If I change to -mlzo1x, it does pack, but only to 100% ratio - no inflation.

[ffmla]

This was an error when I typed the post and not from my arc.ini.

If I change to -mlzo1x, it does pack, but only to 100% ratio - no inflation.
During my testing with Far Cry 3 repack i am also faced this issue but ,I don`t know which version of XTOOL is this.
Use the attached one and try it.

My result with FC3.
8.51GB to 4.04 GB

EDT:XTOOL version 1908_R5

[Masquerade]

Here's another Unity sample, xtool doesn't detect any streams at all (regardless of which liblz4 is used, the previous sample I posted: streams were found, just not processed).

https://1fichier.com/?x630wefm2269eh4d81ya

It decompresses to 6.9GB

[Razor12911]

lzo does not have a universal scanner, for the far cry series, you'll need a database file, like fc3.xtl and then use -mfc3 method.

Here's another Unity sample, xtool doesn't detect any streams at all (regardless of which liblz4 is used, the previous sample I posted: streams were found, just not processed).

https://1fichier.com/?x630wefm2269eh4d81ya

It decompresses to 6.9GB

This contains file version 7, not implemented in unity.dll as it only supports version 6.

An update is required

[Razor12911]

Update available

Changes

- updated search support

[Gamer2023]

What is the best compression method ? how to use it ?

[dixen]

What is the best compression method ? how to use it ?

It's PREcompressor but not compressor

[Gamer2023]

I-How to use it ?
My pc
1.Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz 2.90 GHz
2.8gb ram
II-What is the best compression my computer can handle?
III-Can I use some type of virtual ram to do better compression ?
IV-Can to decompress using inno compiler ?
After compression How could I repack The games?

Sorry for long Questions.:(:(:(:(:(

[st0rm]

I-How to use it ?
My pc
1.Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz 2.90 GHz
2.8gb ram
II-What is the best compression my computer can handle?
III-Can I use some type of virtual ram to do better compression ?
IV-Can to decompress using inno compiler ?
After compression How could I repack The games?

Sorry for long Questions.:(:(:(:(:(

try to use ultraarc.

[Gamer2023]

My pc details Below.
I-How to use it ?
My pc
1.Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz 2.90 GHz
2.8gb ram
II-What is the best compression my computer can handle?
III-Can I use some type of virtual ram to do better compression ?
IV-Can to decompress using inno compiler ?
After compression How could I repack The games?

Sorry for long Questions.:(:(:(:(:(
If I have a game say "Grand Theft Auto San Andreas". How could I repack it ?
Can Anyone tell the process to repack the game to under 1.5 gb ?

[L33THAK0R]

My pc details Below.

If I have a game say "Grand Theft Auto San Andreas". How could I repack it ?
Can Anyone tell the process to repack the game to under 1.5 gb ?

I dunno if 1.5gb is possible for san andreas, at least not for the most current PC release (v1.0.0.22). I was able to get it down to 2.87gb from 4.58gb, with xtool you'll want to use the zlib codec for that title.

[Gamer2023]

I dunno if 1.5gb is possible for san andreas, at least not for the most current PC release (v1.0.0.22). I was able to get it down to 2.87gb from 4.58gb, with xtool you'll want to use the zlib codec for that title.

How ?
Can u tell the complete process ?

[Masquerade]

@Razor12911
Here is another unity sample, it should decompress to 254MB however xtool can only get it to 122MB.

I have tried many liblz4s (including the one you gave me last time). The most recent dll gives the highest size.

https://1fichier.com/?06vig7zj102jjv5l07hz

[Gamer2023]

How to repack a game highly compressed ?
You Tell step by step.(Please provide files used for compression ) I don't know much about repacking.
For example a game alien shooter.
Attachment Removed by Grumpy
Create a repack and provide me. Can you repack it to less than 10 mb.
Please provide the steps.
Setup may be like Fitgirl or normal installation ?

[Grumpy]

How to repack a game highly compressed ?
You Tell step by step.(Please provide files used for compression ) I don't know much about repacking.
For example a game alien shooter.
Attachment Removed by Grumpy
Create a repack and provide me. Can you repack it to less than 10 mb.
Please provide the steps.
Setup may be like Fitgirl or normal installation ?

Do not post full game links.

You need to read/search the forums for the answers you are after, all the information can be found on the forums, you can not expect to come onto a forum and have members take there time to just answer your questions without doing some of your own research first.

[Game Repacker]

I got how to use the xtool. But I want to know the best compression of an 8 GB, 6 GB, 4 GB ram pc can repack ?

[Harsh ojha]

i got how to use the xtool. But i want to know the best compression of an 8 gb, 6 gb, 4 gb ram pc can repack ?

dm 😅

[sajmon83]

I have been trying for a long time to use the latest version of xTool for LZ4 compression. Unfortunately, no results. I did a short test with a sample. Only the older xTool can handle it. My question is, am I doing something wrong? Bad syntax, parameter? I've tried multiple versions of liblz4.dll and unfortunately no result. In the test, I used the latest version of the liblz4.dll v1.9.3 library on all xTool versions. And only one brings results (v0.12). The question is what am I doing wrong?
I enclose a link to the sample and test (https://1fichier.com/?8q192lf45b5c03vg2ozl)

And the contents of the test script
xtool_0.3.03_hotfix\xtool.exe precomp -mlz4 -c1024mb -t100p "sniperescape.ff" xtool_0.3.03sniperescape.ff.unlz4
xtool_0.3.03_hotfix\xtool.exe precomp -mlz4hc -c1024mb -t100p "sniperescape.ff" xtool_0.3.03sniperescape.ff.unlz4hc

xtool_0.3.21\xtool.exe precomp -mlz4 -c1024mb -t100p "sniperescape.ff" xtool_0.3.21sniperescape.ff.unlz4
xtool_0.3.21\xtool.exe precomp -mlz4hc -c1024mb -t100p "sniperescape.ff" xtool_0.3.21sniperescape.ff.unlz4hc

xTool_0.12\xtool.exe e:precomp:c1024mb,t100p:lz4 "sniperescape.ff" xTool_0.12sniperescape.ff.unlz4
xTool_0.12\xtool.exe e:precomp:c1024mb,t100p:lz4hc "sniperescape.ff" xTool_0.12sniperescape.ff.unlz4hc

xtool_2010_R5\xtool.exe precomp -mlz4 -c1024mb -t100p "sniperescape.ff" xtool_2010_R5sniperescape.ff.unlz4
xtool_2010_R5\xtool.exe precomp -mlz4hc -c1024mb -t100p "sniperescape.ff" xtool_2010_R5sniperescape.ff.unlz4hc

xtool_2012_R2\xtool.exe precomp -mlz4 -c1024mb -t100p "sniperescape.ff" xtool_2012_R2sniperescape.ff.unlz4
xtool_2012_R2\xtool.exe precomp -mlz4hc -c1024mb -t100p "sniperescape.ff" xtool_2012_R2sniperescape.ff.unlz4hc

pause

[Game Repacker]

Do you know the best compression method ?

[sajmon83]

Game Repacker One way to find the best compressor is to try it out (or possibly knowledge of the engine and its compression method). One of the more important methods is the precompressor (if there is one). Let me give you a simple example on the CoD 4 MW Remaster. All * .ff files 10870.08Mb I passed through precompressor (most files are already compressed precomporsssor such as xTool can extract these files) xTool_lz4 which gave me a result of 24633.98 Mb Then I used srep (the tool finds the same files and duplicates) which gave me 9113.11 Mb. And for all this I use the LOLZ compressor (currently one of the best compressors for graphics files) which gives me 2,766.74 Mb which gives an amazing 25.45% result. But not every file type gives the best result for LOLZ compression. For some it is a razor compressor for others LZMA and different for audio files. There is simply no one method.

There's a great FitGirl post on the web about compression. I recommend you to find it and read it there is probably everything you need to know about compression these days ^^ And I think that as simply as it was explained there, it cannot be easier ^^ And above all, beautifully written

[Game Repacker]

Game Repacker One way to find the best compressor is to try it out (or possibly knowledge of the engine and its compression method). One of the more important methods is the precompressor (if there is one). Let me give you a simple example on the CoD 4 MW Remaster. All * .ff files 10870.08Mb I passed through precompressor (most files are already compressed precomporsssor such as xTool can extract these files) xTool_lz4 which gave me a result of 24633.98 Mb Then I used srep (the tool finds the same files and duplicates) which gave me 9113.11 Mb. And for all this I use the LOLZ compressor (currently one of the best compressors for graphics files) which gives me 2,766.74 Mb which gives an amazing 25.45% result. But not every file type gives the best result for LOLZ compression. For some, it is a razor compressor for others LZMA and different for audio files. There is simply no one method.

There's a great FitGirl post on the web about compression. I recommend you to find it and read it there is probably everything you need to know about compression these days ^^ And I think that as simple as it was explained there, it cannot be easier ^^ And above all, beautifully written

What code used for xtool?

[sajmon83]

Run xTool in cmd xtool precomp and xtool decode
You will see all the information about the xtool or run xtool.chm which also contains most of the information

[Game Repacker]

I used free arc to compress GTA San Andreas. from 4.5Gb to 3.3 Gb.
Can I use precomp+srep+freearc for a better one?
Which one would be the best process?
Please tell me.
:p:p:p:p

[Game Repacker]

I got the files used by fit girl repacks. I providing files but don't know how to use these files.
These files I found in the temp folder during the installation of a game from the official website.
Files below.

[L33THAK0R]

those are just the files used for unpacking the archive(s), the majority of the tools you need for data compression/decompression can be found here, under the "stickied" / "pinned" posts.

[doofoo24]

@Game Repacker
this is XTOOL THREAD :mad::mad::mad:
there is guide threads on the site how to compress games..

[Game Repacker]

@Game Repacker
this is XTOOL THREAD :mad::mad::mad:
there is guide threads on the site how to compress games..
I know that but usage?

[infovs]

Anyone noticed xtool and unreal.dll plugin are broken for Street Fighter V since xtool 0.3.9 + unrealplugin R3?
Don't know about other games, I own SF5 and did personal repack with 0.3.9 + unreal.dll R3, .pak files are decrypted normally.
Tried to use it on 0.3.21 + unreal.dll R5 -> nothing is decrypted at all

Btw, yes, I used -d1. Maybe I overlooked something, but if that's true can you please explain to me what am I doing wrong.

For sample I used small 20mb pakchunk50-WindowsNoEditor.pak

With xtool.exe 0.3.9 + unreal.dll R3 - WORKING, decrypted, inflated
[External compressor:xtool]
header = 0
packcmd = xtool precomp -mue4:m1:k5F6153346D665A4B384D3573354B5743324C7A325 673466E474B4937617A676C+zlib -d1 -c128mb -t100p --dbase - - <stdin> <stdout>

With xtool.exe 0.3.21 + unreal.dll R5 - NOT WORKING, nothing is decrypted (m1 is not needed in latest unreal.dll version)
[External compressor:xtool]
header = 0
packcmd = xtool precomp -mue4:k5F6153346D665A4B384D3573354B5743324C7A325673 466E474B4937617A676C+zlib -d1 -c128mb -t100p --dbase - - <stdin> <stdout>

Used -mxtool+srep:m2f:l512+lolz arc.exe pack command on both.

[Joe Forster/STA]

I know that but usage?

If you have specific questions then ask them. Otherwise, I quote from the first post of this thread: "Read the documentation to find out how it works and how to use it."

[L33THAK0R]

Does using the "LZ4" codec require a specific library version? I'm currently trying to pack the Sony "PlayStation 3" exclusive, "Demon's Souls" (which has LZ4 streams). Attempting to use xtool 2020 results in this output (https://imgur.com/a/5MTnm88) as soon as the temp file has been generated. Utilising xtool 2019 only inflates the input by ~1kb. I'm not able to find any prior examples of the codec being used so I'm not quite sure whats going on.

[L33THAK0R]

Has anyone encountered this error (https://imgur.com/a/vdye9LV) before? I've had it happen with a handful of game-data files, curiously I was able to pack this title ("DOOM", the 2016 reboot of the series) previously, when using the default scanning range (I think 16mb), but I'm redoing it since some files failed to decompress properly.

[MineRocker]

Has anyone encountered this error (https://imgur.com/a/vdye9LV) before? I've had it happen with a handful of game-data files, curiously I was able to pack this title ("DOOM", the 2016 reboot of the series) previously, when using the default scanning range (I think 16mb), but I'm redoing it since some files failed to decompress properly.

maybe it's an issue with the -t100p-1? :confused:

[Razor12911]

Here's the documentation. It took weeks to write this as I'm very busy.

I'm still unavailable for two more months but the documentation should give insights as to how to fully utilize xtool and how to create plugins.

[PsYcHo_RaGE]

Here's the documentation. It took weeks to write this as I'm very busy.

Thank you very much @Razor12911 👍

[KaktoR]

@Razor12911
Thanks for the documentation. Brought me a few steps further (I didn't even know that xtool has mermaid/leviathan support until I read your documentation lol).
But what I don't understand right now is what to do if there is no signature (or repeating pattern) before an oodle header or if the pattern is different before many headers.

Currently I'm dealing with mermaid (8C 0A) and there is no visible pattern before the header, just always different random bytes.

---

In a different test I was looking for kraken header (8C 06). There seems to be a repeating pattern, namely 6 bytes (00 00 00 00 01 00) + 1 or 2 random byte. The 9th byte is always X (58).

https://i.imgur.com/qEKZ9IK.png

Maybe I have to work with Conditions, but unfortunatelly you forgot to explain them somehow in the documentation :D

[dixen]

LZMA Codec for TTW2. It possible?)

[Razor12911]

maybe i have to work with conditions, but unfortunatelly you forgot to explain them somehow in the documentation :d

30469

:(

lzma codec for ttw2. It possible?)

yes, read documentation and make plugin :rolleyes:

[KaktoR]

@Razor12911
Sorry, I already found it (I just forgot to edit my post) and I made some steps forward (on my sample I made +3% ratio compared to xtool without configutation, I will see if there are more) :)

[Razor12911]

Update available

Changes

- updated search support (speed improvements)
- updated command line parser
- added partial universal scanner for lzo1x streams
- added universal scanner for lz4f streams
- fixed issue with configuration files failing to execute without conditions

Notes

The previous update is still up, in case if there are issues with this one as I'm rusty af at the moment. (I haven't coded in months)
The lzo1x codec uses LZO1X-999 (level 1-9) and should only be used if you have no alternatives for precompression such as no plugins as this thing can be slow at times.

[Masquerade]

Razor12921
Welcome back, thank you for the update. Merry Christmas! :D

[L0v3craft]

Update available

Changes

- added universal scanner for lz4f streams

Thanks Razor! Please can you write a list of games that are compatible with that lz4f scanner? For example is it for new or old unity games?

[panker1992]

the xtool has an xtool.ini with xcompress.exe and crilayla.exe where can i find those ?

[Masquerade]

the xtool has an xtool.ini with xcompress.exe and crilayla.exe where can i find those ?

I think they are coming soon.

[Razor12911]

Update available

Changes

- project made open source
- added external executable support
- added generate database feature
- fixed search support bug

Notes

The source code will be made available on GitHub
https://github.com/Razor12911/xtool

External executable support is still in development and not thoroughly tested and below shows how it is used within xtool.ini

[nier_replicant]
Encode=zstd138.exe -c -d - <stdin> <stdout>
Decode=zstd138.exe -c --fast=1 --long=15 - <stdin> <stdout>
Continous=0

possible constants: <stdin> <stdout> <filein> <fileout> <fileres> <filestore> <size>

[Razor12911]

Update available

Changes

- fixed issue of status not reporting when encoding
- added depth method support for search support
- fixed zlib encoding issues for different window bits (thanks KaktoR)
- fixed zlib memory leak issue (thanks KaktoR)
- updated all internal codecs to support information relayed by external codecs
- updated lz4f codec and removed temporarily removed support for universal scanning
- added option to change recompression level to be used by reflate
- updated external executable support
- generate database feature currently bugged, wait for next update
- search database structure changed, older database files will no longer work with newer releases

Notes

Older database files will not work with future updates, use the older version 0.3.21 for old database files as you wait for them to be updated
A temp file may be created when using database files which can be the size of the largest file in the data to be processed.

[Anonymous0000]

I'm in win11.

[daveyrob]

im curious, if you can now select a reflate level what does it do as standard? maybe set to 6 as as default?

[Razor12911]

I'm in win11.

Xtool isn't used directly, look at test example or the Freearc example to see how it works.

im curious, if you can now select a reflate level what does it do as standard? maybe set to 6 as as default?

The idea of reflate is by selecting the correct level you get the best result. You can perform trial and error on an input by trying out different levels and the best one is the one that gives the least output. Default is level 6

[Razor12911]

Update available

Changes

- removed debugging code from encryption and executable codec
- fixed issue with depth when using search codec
- fixed external executable support issues

[ScOOt3r]

Im trying to understand the commands for xtool_0.4.2

my arc.ini file is this

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+preflate -c32mb -t85p - - <stdin> <stdout>

[External compressor:lolz]
header = 0
packcmd = lolz\lolz_x64.exe -d256 -dt -dtb1 -mtt1 -mt14 -fba1024 -tt5 -mc128 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

[External compressor:srep]
header = 0
packcmd = srep\srep.exe -m3f -l512 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

and the pack.bat file is this
del /q data.bin
arc.exe a -ep1 -r -ed -s; -w.\temp -mxtool+srep+lolz data.bin "pack\*"
pause

it creates a freearc1.tmp file only and not a .bin file
what am i doing wrong with this.

Thanks

[Razor12911]

try removing all plugins (xtl and ini) files near xtool, they may be causing it to crash and try again.

[seryogakms]

ERROR: write error (disk full?) in compression algorithm xtool

arc.ini
[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+kraken:l4 -c256mb -lm - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

[External compressor:srep]
header = 0
packcmd = srep64 -mem512m -l512 -m5f $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

[External compressor:lolz]
header = 0
packcmd = lolz_x64.exe -d128 -mc1023 -dtb1 -tt4 -mtt1 -mt8 -mtb128 -fba1024 -oh14 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = lolz_x86.exe $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

cmd:
arc.exe a -ep1 -r -ed -s; -i2 -w.\temp -mxtool+srep+lolz Packs\data.bin "data\*"

With xtool_0.3.9 there is no such problem.

What could be the problem?

Thank you in advance.

[ScOOt3r]

I did try and remove all .xtl and .ini files and same thing.. just a freearc1.tmp was created only

ScOOt3r

[Masquerade]

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+kraken:l4 -c256mb -lm - - <stdin

zlib+kraken? :confused:

[Anonymous0000]

Is there any ui-based interface that I can use to input the commands?

[ScOOt3r]

i just used the same files located in xtool 0.4.2
and here are my pics

31263
It compressed just fine with no errors

31264
here are the files for xtool 0.4.2 as you can see it created a freearc1.tmp file

31265
here are the files that i was test packing

my arc.ini is
[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib+preflate -c32mb -t85p - - <stdin> <stdout>

[External compressor:lolz]
header = 0
packcmd = lolz\lolz_x64.exe -d256 -dt -dtb1 -mtt1 -mt14 -fba1024 -tt5 -mc128 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

[External compressor:srep]
header = 0
packcmd = srep\srep.exe -m3f -l512 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

my pack.bat file is
arc.exe a -ep1 -r -ed -s; -w.\temp -mxtool+srep+lolz data.bin "pack\*"

[seryogakms]

zlib+kraken? :confused:

https://imageup.ru/img261/3886963/2022-02-24_054012.png

[Razor12911]

Is there any ui-based interface that I can use to input the commands?

https://fileforums.com/showthread.php?t=104507

zlib+kraken? :confused:

I doubt that's the issue, @ScOOt3r only uses zlib related codecs however he has an issue of his own (might be related)

@seryogakms, @ScOOt3r

I'll take a look at it but it will be challenge because I am not experiencing any of the issues you're reporting.

[Hexagon123]

When using Knight Compressor, it uses too much DLL files that made it crash. But, I deleted some to make it work. Multiple files work but not Unity.

[Masquerade]

seryogakms
Apologies for criticising your zlib+kraken method, I have tested out a new game which contains both. Sorry. Hope you can forgive.

[Razor12911]

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

@seryogakms, @ScOOt3r
can you try this configuration, if xtool throws any errors please post the error message here

[ScOOt3r]

ok after testing it i now get this error

if i do a clean install of xtool 0.4.1 with no library files added i get this

it creates the data.arc with this xtool setting, but it wont decompress with out errors.


[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

[Razor12911]

can you share this input along with your system specs?

[ScOOt3r]

My System Specs are as follows:
AMD Ryzen 9 5950X
16 Cores
32 Logical Processors
64 GB of Ram
NVIDIA GeForce RTX3090 GPU

[daveyrob]

i had the same Disk Full issue, remove libzstd.dll and it will work.

[seryogakms]

seryogakms
Apologies for criticising your zlib+kraken method, I have tested out a new game which contains both. Sorry. Hope you can forgive.

It's all right!

[BLACKFIRE69]

ok after testing it i now get this error

if i do a clean install of xtool 0.4.1 with no library files added i get this

it creates the data.arc with this xtool setting, but it wont decompress with out errors.


[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>


ScOOt3r,
this is ISDone right? how about the unpack.bat? you get the same error?

[ScOOt3r]

well i got it working compressing and decompression as someone posted above to remove libzstd.dll.. i did that and it works just fine now.. weird that it was that file
not sure how important libzstd.dll is...

ScOOt3r

[seryogakms]

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

@seryogakms, @ScOOt3r
can you try this configuration, if xtool throws any errors please post the error message here

File Patch.bin from "Horizon zero dawn" (last version).
5 434 479 154 bytes → 360 912 358 bytes (xtool_0.3.9 ONLY). Compression time — 36 minutes. Unpacking time — 18 minutes.

Intel core i5-3550, 20 GB RAM.

arc.exe a -ep1 -r -ed -s; -i2 -w.\temp -mxtool+srep+lolz Packs\data.bin "H:\06. 1.0.11.9 (1.11)\Packed_DX12\Patch.bin"

[External compressor:xtool]
header = 0
packcmd = xtool precomp -mkraken -d3 -c64mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

[External compressor:srep]
header = 0
packcmd = srep64 -mem512m -l512 -m5f $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

[External compressor:lolz]
header = 0
packcmd = lolz_x64.exe -d128 -mc1023 -dtb1 -tt4 -mtt1 -mt8 -mtb128 -fba1024 -oh14 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = lolz_x86.exe $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

No other version of xtool gives such results under any settings.

xtool_0.4.2 with the same (and other) settings:
5 434 479 154 bytes → 2 738 426 574 bytes. Compression time — 3 hours and 56 minutes. Unpacking time — 18 minutes.

[dixen]

File Patch.bin from "Horizon zero dawn" (last version).
5 434 479 154 bytes → 360 912 358 bytes (xtool_0.3.9 ONLY). Compression time — 36 minutes. Unpacking time — 18 minutes.

Intel core i5-3550, 20 GB RAM.
oo2reck+srep+lolz - 340 mb
Decompression time - 4 min:cool:

[Razor12911]

File Patch.bin from "Horizon zero dawn" (last version).
5 434 479 154 bytes → 360 912 358 bytes (xtool_0.3.9 ONLY). Compression time — 36 minutes. Unpacking time — 18 minutes.

Intel core i5-3550, 20 GB RAM.

arc.exe a -ep1 -r -ed -s; -i2 -w.\temp -mxtool+srep+lolz Packs\data.bin "H:\06. 1.0.11.9 (1.11)\Packed_DX12\Patch.bin"

[External compressor:xtool]
header = 0
packcmd = xtool precomp -mkraken -d3 -c64mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

[External compressor:srep]
header = 0
packcmd = srep64 -mem512m -l512 -m5f $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

[External compressor:lolz]
header = 0
packcmd = lolz_x64.exe -d128 -mc1023 -dtb1 -tt4 -mtt1 -mt8 -mtb128 -fba1024 -oh14 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = lolz_x86.exe $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

No other version of xtool gives such results under any settings.

xtool_0.4.2 with the same (and other) settings:
5 434 479 154 bytes → 2 738 426 574 bytes. Compression time — 3 hours and 56 minutes. Unpacking time — 18 minutes.

Sample?

[seryogakms]

well i got it working compressing and decompression as someone posted above to remove libzstd.dll.. i did that and it works just fine now.. weird that it was that file
not sure how important libzstd.dll is...

ScOOt3r

https://imageup.ru/img160/thumb/13887671.jpg (https://imageup.ru/img160/3887671/1.png)

arc.exe a -ep1 -r -ed -s; -i2 -w.\temp -mxt22+srep+lolz Packs\data.bin "DATAS\*"

[External compressor:xt22_zstd]
header = 0
packcmd = xtool\zlib\xt22.exe precomp -mzlib+zstd:l19,l22 -c32mb -t100p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = xt22.exe decode -t100p - - <stdin> <stdout>

[External compressor:lolz]
header = 0
packcmd = lolz_x64.exe -d128 -mc1023 -dtb1 -tt4 -mtt1 -mt8 -mtb128 -fba1024 -oh14 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = lolz_x86.exe $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

[External compressor:srep]
header = 0
packcmd = srep64 -mem512m -l512 -m5f $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

[seryogakms]

oo2reck+srep+lolz - 340 mb
decompression time - 4 min:cool:

В очередной раз спасибо за подсказку, но у меня комп «немного» слабее твоего. ):

[dixen]

В очередной раз спасибо за подсказку, но у меня комп «немного» слабее твоего. ):
shhh. english only in post...
Razor12911
Hi)
What about *.bdt from Elden Ring? It encrypted oodle streams?

[seryogakms]

shhh. english only in post...

Sorry, I didn't think of that...

[Razor12911]

shhh. english only in post...
Razor12911
Hi)
What about *.bdt from Elden Ring? It encrypted oodle streams?

sample?

[dixen]

sample?
Smallest bdt is 2.4 gb..Uploading..

https://drive.google.com/file/d/1UDbxidyWSfnT6RsrDZ6mQZon0TIIxDk9/view?usp=sharing

[seryogakms]

Sample?

It took even longer. I apologize for the encoding in cmd.

xtool_0.4.2

https://imageup.ru/img119/thumb/113887693.jpg (https://imageup.ru/img119/3887693/11.png)

[:( Sad8669]

ELDEN RING

Oodlerec Used
Compressing 1 file, 2,655,357,776 bytes
Compressing Data3.bdt
Compressed 1 file, 2,655,357,776 => 4,549,448,231 bytes. Ratio 171.33%
Compression time: cpu 3.81 sec/real 615.49 sec = 1%. Speed 4.31 mB/s
All OK

From Masquerade.
use oodle core from Martha is Dead

[Razor12911]

@dixen

use oo2core_8_win64.dll

31310
31311

It took even longer. I apologize for the encoding in cmd.

xtool_0.4.2

https://imageup.ru/img119/thumb/113887693.jpg (https://imageup.ru/img119/3887693/11.png)


upload the file from the game for me to take a look at.

[Masquerade]

You can use this oodle core to boost the ratio a little bit more
https://bayfiles.com/55bfieKbx4/oo2core_4_win64_7z
Credits to FitGirl for the dll

[KaktoR]

From a 100mb sample data0.bdt

xtool:kraken
Compressed 1 file, 104,857,600 => 213,800,403 bytes. Ratio 203.90%
Compression time: cpu 0.09 sec/real 15.75 sec = 1%. Speed 6.66 mB/s

xtool:kraken,l6
Compressed 1 file, 104,857,600 => 214,235,226 bytes. Ratio 204.31%
Compression time: cpu 0.09 sec/real 10.01 sec = 1%. Speed 10.48 mB/s

[seryogakms]

upload the file from the game for me to take a look at.

LINK ON FILE (https://drive.google.com/file/d/1iR13iLdGgEyq4oGeveXNMMdvVY8kqGFe/view?usp=sharing)

xtool_0.3.9 ONLY
https://imageup.ru/img34/thumb/23887702.jpg (https://imageup.ru/img34/3887702/2.png)

Thanks to DIXEN (https://fileforums.com/member.php?u=217082) for the hint.

UPD:
https://imageup.ru/img242/thumb/3333887736.jpg (https://imageup.ru/img242/3887736/333.png)

https://imageup.ru/img193/thumb/4443887752.jpg (https://imageup.ru/img193/3887752/444.png)

[Razor12911]

Update available

Changes

- added verbose mode
- added feature that allows you to enforce a different library to be loaded
- fixed issues related to imperfect stream patching
- fixed issues with old libraries with missing functions that cause xtool to crash on startup
- updated oodle codec
- updated reflate codec
- updated zstd codec

Notes

You can now enforce xtool to use a specific library rather than renaming the library for it to picked up by xtool.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp { -moption} -c32mb -t100p --zstd=libzstd148.dll - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - --zstd=libzstd148.dll <stdin> <stdout>

Available parameters for this are:
--zlib=filename, --lz4=filename, --lzo=filename, --zstd=filename, --oodle=filename

You can also use this to run multiple tests to see which zstd or oodle library gives the best output by running these commands on a small sample:
xtool.exe precomp -mkraken -c32mb -t100p --oodle=oo2core_7_win64.dll - - < %1 > %1.out1
xtool.exe precomp -mkraken -c32mb -t100p --oodle=oo2core_7_win64_2.dll - - < %1 > %1.out2
xtool.exe precomp -mkraken -c32mb -t100p --oodle=oo2core_8_win64.dll - - < %1 > %1.out3
xtool.exe precomp -mkraken -c32mb -t100p --oodle=oo2core_fifa22.dll - - < %1 > %1.out4

Then you can pick whichever produces the best results

___

Oodle code has been updated, if you having issues with stream detection you can use the parameter n1 (kraken,n1), where the old code will be used. it can also be n2 or n3, depending on how many times each stream should be redetected (for improved results but at a cost of speed)
If this doesn't help then use the side project.

___

Verbose mode was added to give information about input, as to what parameters are used for streams within the input and if you see a common trend then you directly use these parameters to speed up xtool and also give it the opportunity to patch streams if there are any invalid streams.

Verbose mode will enforce xtool to 1 thread thus it should not be used by default as it will slow now precompression.

[KaktoR]

How to use verbose mode

@echo off
xtool.exe precomp -mcodec -c32mb -t100p --zlib=filename, lz4=filename, lzo=filename, zstd=filename, oodle=filename --verbose - - < %1 > %1.out1
pause
xtool.exe decode -t100p - - < %1.out > %1.res
del /q %1.out
fc /b %1 %1.res
del /q %1.res
pause

Drag&Drop a file on batch

[:( Sad8669]

ELDEN RING - XTool Update Test (Data3.bdt)

Old XTool - xtool:kraken,l6
Compressed 1 file, 2,655,357,776 => 4,668,458,345 bytes. Ratio 175.81%
Compression time: cpu 4.09 sec/real 673.02 sec = 1%. Speed 3.95 mB/s
All OK

New XTool- xtool:kraken,l6
Compressed 1 file, 2,655,357,776 => 4,551,696,149 bytes. Ratio 171.42%
Compression time: cpu 3.72 sec/real 564.25 sec = 1%. Speed 4.71 mB/s
All OK

New XTool - xtool:kraken,l6,n1
Compressed 1 file, 2,655,357,776 => 4,673,342,606 bytes. Ratio 176.00%
Compression time: cpu 4.59 sec/real 706.91 sec = 1%. Speed 3.76 mB/s
All OK

New XTool - xtool:kraken,l6,n2
Compressed 1 file, 2,655,357,776 => 4,673,136,986 bytes. Ratio 175.99%
Compression time: cpu 4.05 sec/real 672.64 sec = 1%. Speed 3.95 mB/s
All OK

Library Used : oo2core_4_win64.dll

Check previous page and download the library from there.

[Gehrman]

resident evil 4 HD project (https://www.re4hd.com/) "*.lfs" Precompressor
https://github.com/emoose/re4-research

Arc.ini
[External compressor:re4-lfs]
header = 0
solid=0
packcmd = re4lfs.exe files.lfs files.pack
unpackcmd = re4lfs.exe files.pack files.lfs
datafile = files.lfs
packedfile = files.pack

https://upload.cc/i1/2022/02/28/aWcFp5.jpg


--------------------------------------------------------
But it does not work with XTool
XTool.ini
[re4lfs]
Encode=re4lfs.exe files.lfs files.pack
Decode=re4lfs.exe files.pack files.lfs
Continous=0
https://upload.cc/i1/2022/02/28/BdNR6z.jpg



Is there a way to solve it?

[Razor12911]

can you send sample and the exe?

[dixen]

Razor12911
Is it possible to make lz2k-plugin for XTool?

[Masquerade]

Razor12911
Is it possible to make lz2k-plugin for XTool?

There's no recompression code for this algorithm, not even QuickBMS can compress LZ2K.

[Gehrman]

can you send sample and the exe?

https://github.com/emoose/re4-research/releases/download/0.1/re4-research_build-0.1a.zip

[Dark0226]

How to decompress unity files? What dlls are needed? And if possible examples?

[Hexagon123]

How to decompress unity files? What dlls are needed? And if possible examples?

Some use 1.8.0 version of liblz4.dll

SMBBHD works like XTool v12, use -munity,lz4hc,l10 (liblz4.dll - 1.8.0)
Cris Tales, has some LZMA doesn't work due to use of LZ4, use UBT
Sense: ACPGS, has some LZMA doesn't work due to use of LZ4, use UBT
SMBBM, has no effect
Trollhunters, use Unity plugin with -munity,lz4hc,l12 (liblz4.dll - 1.9.3)
Tin & Kuna, not much, use UBT
NSLT, possibly that much unlike XTool v12

[Masquerade]

I understand the Leviathan codec isn't to be used directly and instead by plugins, so I am attempting to create a database for Wolfenstein Youngblood using a modified BMS script.

The script dumps the leviathan streams to disk and that should be enough to generate a DB, however I recieve an error that the filename of the database I made is not a valid method if I attempt to precompress.

I have attached below a sample file + BMS script if anyone would like to have a look.

The leviathan stream is signified by the 0x8C0C header.

https://www43.zippyshare.com/v/fCNMmXpB/file.html

[elit]

Thank you for last update, it solved zstd crash for me.
Would it be possible for xtool to additionally print its version when run without parameters? So that it does not rely on file name.

Also, additionally to direct dll loading, I would suggest automatic multi lib query like this:
1_zstd.dll, 2_zstd.dll...
1_lz4.dll, 2_lz4.dll...
1_oodle.dll, 2_oodle.dll...

With those files in xtool directory, *all* of them would be loaded and then sequentially tried up to certain point.
For example at the beginning all compression variation on all dll's would be tried until at least 1 valid chunk is found. Then dll's without result would be discarded and if multiple exist with different size then those with lower size also discarded. Those with same size could still be used up to certain number of chunks in parallel unless either different size is produced or simply all but 1 are discarded in case all kept producing same size.

This way user can just put all different versions of libraries and forget about manual "query and try".

In case different versions use different API, another naming alternative would be agreed naming convention, for example zstd_v1.2.dll, lz4_v1.2.dll or any other, where xtool would be aware of that version(or version range) and which names of API functions specific dll use.

[Razor12911]

Update available

Changes

- fixed issue of lz4 codec loading incorrect library
- fixed issue with handling endianess via configuration based plugins
- updated framework of library based plugins

@elit
For example at the beginning all compression variation on all dll's would be tried until at least 1 valid chunk is found.
- This cannot be done because versions closer to each other tend to produce the same result and in some streams especially on a small number of them, they produce different results.
- Example, version 1.4.3 and 1.4.4 of zstd, these two libraries can produce same result and would be considered valid chunk using either one of these libraries however this may not be the same for the rest of the input.

Those with same size could still be used up to certain number of chunks in parallel unless either different size is produced or simply all but 1 are discarded in case all kept producing same size.
- The issue with this is considering xtool processes data by chunks, if it processes version by version, it would need to store all results somewhere, most probably in memory and every single library would need its own memory set, coding this would be problematic at least from my end.

@Gehrman

Sorry for late reply, I checked out the sample and the only way this would work is by generating a database using generate function in xtool (which is currently broken), but it will probably be fixed in the next release.
I'll possibly upload the re4lfs configuration example. :)

[dixen]

Thank for update..
But..Metro Exodus lz4 streams still not being searched for((

[Razor12911]

Metro Exodus lz4 support was never added to xtool, remember you cannot use lz4 codec directly in this program, only via plugins and a plugin for that game was never made.
https://fileforums.com/showthread.php?t=103563

[Razor12911]

Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

-mleviathan
If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.

[elit]

Razor I think you mentioned in the past that -mzlib is able to detect deflate? I recall mentioning back then that it did not work for me. So here I am attaching set of png images that I just accidentally tested and did not work. Both -mreflate as well as previous ztool:high does work.

31371

[Razor12911]

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

XTool is created by Razor12911

[0] Performing scan from block 0000000000000000 to 000000000000A046 (41031)
[0] Actual zlib stream found at 0000000000000053 (40928 >> 87680)

[0] Processing streams on block 0000000000000000 to 000000000000A046 (41031)
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l11,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l12,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l13,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l14,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l15,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l16,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l17,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l18,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l19,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l21,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l22,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l23,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l24,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l25,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l26,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l27,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l28,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l29,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l31,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l32,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l33,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l34,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l35,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l36,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l37,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l38,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l39,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l41,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l42,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l43,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l44,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l45,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l46,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l47,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l48,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l49,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l51,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l52,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l53,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l54,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l55,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l56,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l57,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l58,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l59,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l61,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l62,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l63,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l64,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l65,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l66,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l67,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l68,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l69,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l71,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l72,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l73,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l74,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l75,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 3072) using l76,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l77,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l78,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l79,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l81,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l82,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l83,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l84,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l85,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l86,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l87,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l88,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l89,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l91,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l92,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l93,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l94,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l95,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l96,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l97,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l98,w15 has failed
[0] Processing zlib stream at 0000000000000053 (40928 >> 87680 >> 512) using l99,w15 has failed

Streams: 0/1
Time: 00:00:00 (00:00:00)
Memory: 128 MB (128 MB)

I tried to precompress one of the png files and as you can see, the method -mzlib was able to detect the deflate stream but then it failed to process it.

[KaktoR]

Update available

Changes

- removed leviathan codec restriction

Notes

Xtool can "actually" detect leviathan streams however the only way it can process them is if they are divisible by the block size used by the new oodle compressors which is 262144, if a stream decompressed size cannot be divided by this value leaving no remainder then no way of predicting the stream size (at least that I know of) is possible hence why plugins are the only ones that were allowed to use the leviathan codec.

-mleviathan
If you are not satisfied with the results produced then find a way to make a plugin for proper support for that particular game.

It's better then nothing I guess, especially if you're not familiar with creating plugins, so thanks for the update :D

Streams: 354/415
Time: 00:01:18 (00:01:17)
Memory: 423 MB (423 MB)

64,1mb >> 352mb

[Masquerade]

Thanks Razor, this will be good for when the generator is working.

[elit]

-mzlib can detect deflate streams, but similarly to my previous post regarding leviathan, leviathan streams are detected however detection and being able to process them are two different things.

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

[elit]

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

31395

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
31394

PS(could it be due to window size - 48k ?)

[Razor12911]

https://github.com/emoose/re4-research/releases/download/0.1/re4-research_build-0.1a.zip

Sorry for late response but here's an example.

in the generate folder, you'll get the example of how database would be created.

xtool.exe generate -mre4lfs "game\*" "game\*" lfs.xtl

You'll notice that the unpacked data and the original data is the same, that's because this tool accepts game files as they are and doesn't need to extract additional streams.

Note: Chunk size is should be the same size as the largest stream/file

then in precompress folder is where you need to transfer the generated database file from generate folder to precompress folder (lfs.xtl)

Look in xtool.ini to see how the program should be configured based on the information you provided.

[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs

Results of the samples you provided
Compressed 5 files, 2,853,066 => 6,511,766 bytes. Ratio 228.24%
Compression time: cpu 0.02 sec/real 1.06 sec = 1%. Speed 2.70 mB/s

[Razor12911]

Thanks, so on more technical ground what is to be expected going forward regarding this? Specifically why is old reflate able to work without issues and mzlib not, is it matter of time/rework or is this a implementation limitation that is to be expected from mzlib, as compare to reflate and will not change in future?

there are several implementations of the deflate algorithm, it's very flexible. Think of using lzma to compress something, obviously there are compression levels that it comes with 1-9. -mzlib implementation is based around this idea that a user who uses this would have used a common setting such as compression level so for precompression it will decompress the data and then try to recompress it using these common levels, but of course lzma has methods that people use for other purposes like for stronger compression such as "lzma:ultra:a1:mfbt4:d158m: fb273:mc1000000000:lc8" mzlib cannot process such, similarly to deflate streams in png images as the way deflate set in such a way that it's good compressing pixels of an image but also make it quicker for loading/viewing. Reflate and preflate have no problem processing these as they used an advanced method for precompressing them.

You might say that mzlib is therefore useless, actually it still has its uses. mzlib is faster than both reflate and preflate because of its simple implementation and incurring no overhead furthermore mzlib produces better results because there is no additional header information file produced hence why I've preached several times to mix both zlib with reflate or preflate, if zlib fails to process the stream it just passes the stream to reflate/preflate for it to process it.

mzlib:
Compressed 1 file, 86,347,072 => 444,743,922 bytes. Ratio 515.07%
Compression time: cpu 0.16 sec/real 13.37 sec = 1%. Speed 6.46 mB/s

mreflate,l9:
Compressed 1 file, 86,347,072 => 445,179,589 bytes. Ratio 515.57%
Compression time: cpu 0.14 sec/real 14.33 sec = 1%. Speed 6.03 mB/s

mpreflate:
Compressed 1 file, 86,347,072 => 366,023,129 bytes. Ratio 423.90%
Compression time: cpu 0.14 sec/real 10.25 sec = 1%. Speed 8.42 mB/s

mzlib+reflate,l9:
Compressed 1 file, 86,347,072 => 444,941,384 bytes. Ratio 515.29%
Compression time: cpu 0.06 sec/real 11.74 sec = 1%. Speed 7.35 mB/s

mzlib+preflate:
Compressed 1 file, 86,347,072 => 444,744,743 bytes. Ratio 515.07%
Compression time: cpu 0.11 sec/real 11.23 sec = 1%. Speed 7.69 mB/s

As you can see, zlib perform faster when alone but better when you hook it up with reflate/preflate and preflate fails to process some streams. I've given people options, all up to them how they use xtool. :)

I cannot inflate zst(zstd) file from Paper Mario Origami King with xtool. With zstd.exe it does unpack just fine:

31395

I was so desperate that I actually downloaded and compiled ALL available zstd versions from github! I am attaching those dll's, zstd.exe and a .zst file itself:
31394

PS(could it be due to window size - 48k ?)

use --verbose mode to see which levels are producing the closest result to the original stream.
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 17101) using l1 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16535) using l2 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 16351) using l3 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15784) using l4 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15711) using l5 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15493) using l6 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l7 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15462) using l8 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15444) using l9 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15442) using l10 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15438) using l11 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15437) using l12 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 15428) using l13 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14817) using l14 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14733) using l15 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14689) using l16 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l17 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l18 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l20 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l21 has failed
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l22 has failed

level 17 is close but level 19 is commonly used, then set -mzstd,l19
[0] Processing zstd stream at 0000000000000000 (14692 >> 48079 >> 14688) using l19 has failed
[0] - Patched stream at 0000000000000000 (14692 >> 14688) [28] successfully

Compressed 1 file, 14,692 => 48,239 bytes. Ratio 328.34%
Compression time: cpu 0.00 sec/real 0.55 sec = 0%. Speed 0.03 mB/s

Used libzstd130.dll

[elit]

level 17 is close but level 19 is commonly used, then set -mzstd,l19
Thank you Razor! But wait, why it did not tried that level automatically? Is xtool trying only some levels here as well? Like up to 9, maybe due to speed?

If that is the case, may I suggest that it try to do all of them - up to certain mb of data(user defined). For instance first 100-200mb all levels and if nothing found drop to only speedier levels as is now etc.?

[Razor12911]

Dude it tried all levels, look at the log via verbose (level 1-22) and none of them produced crc perfect stream and you can even see that with level 19 the original stream was 14692 which was decompressed to 48079 and recompressed to 14688.

Xtool applies patches using xdelta only if you have specified what level was used else for zstd as an example which has 22 levels, each stream would have to be recompressed 22 times and 22 patches will exist and then how would xtool decide what level was used, by looking at which one produced the smaller diff file? because if you look at the log again level 17, 18, 19, 20, 21 and 22, yep 6 different levels produced the same result so how would xtool decide what the best level is? Even if you do it for the first 100-200mb, you may even find that zstd compression on some data even taps out at level 10 and any level higher than this will produce the same result. Xtool will still be confused af as to which level is the best which is why I left it to the user.

Xtool in such a scenario will just trust the user specified level and only help out if any of the streams fail to be properly processed by applying xdelta otherwise it will not apply any patches.

Edit: Verbose mode was added just so that the user can see why xtool fails and if there are patterns, like for example this stream "14692 >> 48079 >> 14688", with 4 bytes difference, I assume that all the streams in this game will show a pattern of 4 bytes also missing, which can mean when the game was compressed the developers turned on checksum mode for zstd, meaning a hash sized 4 byte was added for all the streams just to verify if the data is not corrupt in anyway and xtool uses zstd in default mode meaning checksum option is disabled hence why they produce 4 bytes less.

[Razor12911]

Note: Chunk size is should be the same size as the largest stream/file


generate -c## ;)

[Gehrman]

generate -c## ;)

good work
@echo off
xtool.exe generate -mre4lfs -c512mb "game\*" "game\*" lfs.xtl
pause

Thank you for this wonderful tool

[elit]

@Razor now I understand, thank you! Its clear that internal knowledge of xtool is still important. All make sense now although it also means my "lz scan" cannot be a simple fire & forget proof for all corner cases(well it could but I feel it would be too much to go with each level separately within the script + all the lib versions etc., maybe compiled binary with proper output report would do). It is fine though, as long as users are aware of this.

-------------------------------------------------------------------------------------------------------------------------------

Unrelated to above, I want to give one advice to all users regarding zstd(possibly lz4 as well). For best results it may be important to try all lib versions even if you found one working. During my switch games repacking I found for example that nearby versions and sometimes far versions may get better result(or pickup again) and by results not only I mean size but also speed. ZSTD have a habit to slow down to crawl(~1mb/s) when used on wrong lib version or can have as much as 2-3x speed difference on nearby versions where all of them can inflate very close to!

I dont have data to show you anymore, but just to imagine it was something like this. Lets say we have a random file of 8Mb, then:
libzstd130.dll > 10000k > 6min
libzstd131.dll > 13000k > 5min
libzstd132.dll > 16000k > 3min
libzstd133.dll > 16400k > 1min
libzstd134.dll > 16500k > 20s
libzstd135.dll > 16550k > 15s
libzstd136.dll > 8000k > 4min (back to original size)
... (here all following versions did nothing)
libzstd144.dll > 16200k > 1min
libzstd145.dll > 16620k > 13s
libzstd146.dll > 8700k > 4min

This is just example, not exact but yes something like that actually happened.

[elit]

Maybe I found a way to pinpoint zstd level. After you scan with 'verbose' you can see there are several levels same. However, using ':l17' to ':l22'(without 'verbose') separately further reveal which level is real. Only with l19 it shows this:
31399
All other levels remain '0/1' even if they show same repacked size in '--verbose' scan. After this it's only matter of figuring out lib version. In this case versions working were 114, 120, 130 and 131. Probably on bigger files time would start showing difference, but I have seen these giving same result before(but also not always) so once they match, any of them should be safe.

[dixen]

[re4lfs]
Encode=re4lfs.exe <filein>.lfs <fileout>.pack
Decode=re4lfs.exe <filein>.pack <fileout>.lfs

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

[Masquerade]

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

Afaik no, because DELZORec requires non solid + 2 input files whereas FA can only deliver 1 input file.

[Razor12911]

Is it possible to implement this with DELZOREC? I mean Far Cry 5
For example

[fc5dec]
Encode=dlzo.exe u <filein>.fat <filein>.dat <fileout>.pack
Decode=dlzo.exe r <filein>.pack <fileout>.dat <fileout>.fat

no, it's not possible because xtool can't be made to give two inputs nor accept two outputs

[Razor12911]

Update available

Changes

- generate database feature fixed
- fixed external executable support issues
- fixed lz4f level setting bug

[Razor12911]

Update available

Changes

- updated oodle scanner
- updated external executable support
- updated configuration based plugin support to add depth information
- updated verbose mode

Notes

Given that very small number of people are able to make their own plugins and codecs to use in xtool, I've added an example that imports codecs from QuickBMS after you have generated your database and have no codec to use with it.

For compression:
[cpk,snappy,rfpk]
Encode=quickbms.exe -s "comtype <codec> ; clog <fileout> 0 <insize> <outsize>" "" <filein>
Decode=quickbms.exe -s "comtype <codec>_COMPRESS ; clog <fileout> 0 <insize> <outsize>" "" <filein>

For encryption:
[blowfish,xxtea]
Encode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 0 <ressize>; log <fileout> 0 <insize>" "" <filein>
Decode=quickbms.exe -s "open FDDE <fileres> 1 ; getdstring X_KEY <ressize> 1; encryption <codec> X_KEY "" 1 <ressize>; log <fileout> 0 <insize>" "" <filein>

You'll need to check http://aluigi.altervista.org/papers/quickbms.txt for the list of supported compressors built in so that you don't pick something like lz2k and expect it to work (quickbms itself has to be able to both compress and decompress). You can add more codecs like I have done here "cpk,snappy,rfpk", if you wanted to add lzss for example, it then becomes "cpk,snappy,rfpk,lzss" then from there you can use Bms2Xtl for scripts that have this comtype and possibly expect xtool to use quickbms with no real issues (hopefully).

Also note that if for whatever reason there is crc mismatch, xtool will still help you by applying xdelta automatically. :)

Also also note that this generate temps files meaning the process is slower because it has to both read and write to disk therefore, only use this approach if there is nothing else that you can to do to improve the situation.

[elit]

OMG ok with this I think you opened can for a whole lot of more codecs/games to be able to process.

[Razor12911]

Update available

Changes

- fixed issue with storing correct recompression information when stream patching is performed

Notes

This issue was brought up by Masquerade and L0v3craft when they were trying to precompress WRC 10 and upon decoding xtool would produce an error so thanks to them, it has been resolved however I'd like to add more useful information regarding this game and why even with xtool helping you with deltas will not allow you to process all streams based on how xtool was designed.

We all know that xtool patches streams if they cannot be restored correctly however it sets its own boundaries, by default the patch file cannot be 5% larger than the original size else the patch fails as highlighted here with some streams being missed. https://fileforums.com/showpost.php?p=496278&postcount=248

This 5% can be changed by user via the command --diff=5p, with 5p being 5%, so to capture all these other streams, you'll have to increase this percentage.

CHUNK_57.PKG
using -mwrc10
Compressed 1 file, 99,435,664 => 189,185,421 bytes. Ratio 190.26%
Compression time: cpu 0.14 sec/real 3.71 sec = 4%. Speed 26.82 mB/s

using -mwrc10 --diff=20p
Compressed 1 file, 99,435,664 => 263,029,277 bytes. Ratio 264.52%
Compression time: cpu 0.09 sec/real 3.61 sec = 3%. Speed 27.53 mB/s

Verbose information displays
[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed
[0] - Patching stream at 0000000000005476 (514 >> 514) [26] has failed

Patching this 514 byte stream produced a 26 byte patch file, which is 5.1% and that is larger than the 5% limit so increasing this means you are allowing xtool to accept this stream.

--diff=20p

[0] Processing lz4f stream at 0000000000005476 (514 >> 893 >> 514) using l9:b4:d0 has failed
[0] - Patched stream at 0000000000005476 (514 >> 514) [26] successfully

[Masquerade]

LEGO® Star Wars™: The Skywalker Saga uses Kraken

XTool is created by Razor12911

Streams: 284071/284071
Time: 00:13:43 (00:48:35)
Memory: 512 MB (512 MB)

100.0%
Errorlevel=0

Compressed 1 file, 4,061,948,534 => 9,598,998,234 bytes. Ratio 236.32%
Compression time: cpu 4.22 sec/real 1032.25 sec = 0%. Speed 3.94 mB/s
All OK

[GamerfromPoland]

OMG it's a great tool! Thanks for it!
https://radaryonline.pl/gdzie-jest-burza/ (https://radaryonline.pl/gdzie-jest-burza/)

[Razor12911]

Update available

Changes

- added IO functions (erase, replace)
- fixed external executable support bugs

Notes

Xtool has introduced IO functions, which should help you perform file and folder operations such as erase and replace (for now, more will be added). I actually wanted to add these functions a long time ago as they could be useful for repacking however I delayed them time after time because precompression needed more attention but as there have been no bug reports for the past 3 weeks I decided to start working on them.

To summarise, Erase is a feature that fills a given input with zeroes in case you wanted to repack certain data separately and Replace is a feature that replaces existing data within certain files with another.
Xtool will search for locations of the extracted streams and store these positions for when you use decode function to revert the changes (yes, the process is reversible)

Erase
xtool erase extracted_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data

Use cases for erase is the removal of language/videos files from archives in games after using another extraction tool, so rather than manually searching for positions and storing them, erase will make this process automated.

An example is after extracting video files from an archive and then wanting to remove credits video, the syntax would be

xtool erase credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak credits.xtl
xtool decode credits.xtl credits.bk2 Gobi\Content\Paks\pakchunk33-WinGDK.pak

decode will fill the zeroes with the original data

Replace
xtool replace old_streams new_streams original_data [decode_data]
xtool decode decode_data extracted_streams original_data

The use cases for replace is possibly game file modification.

An example is having several modified files and the original files and you wanted to bulk replace these files, you'd have to prepare two folders with the same file structure and the syntax would be

xtool replace "textures\original\*" "textures\modified\*" "game\*" mod_tex.xtl
xtool decode mod_tex.xtl "textures\original\*" "game\*"

decode will fill the modified data sectors with the original to restore the file its original form.

PS

If you are using these features for a repack in which people would select languages or video credits for example, if the users did not select any of these to be installed then there would be missing files. Xtool's decode command would still work and will try to restore the original data using whatever data that was selected and available without problems.

[Razor12911]

Update available

Changes

- added IO functions (find, extract, patch)
- generate database feature and IO functions now can search for streams larger than chunk size

Notes

More IO functions introduced. Find simply helps you track positions of extracted files from a given input, while extract uses a generated decode file data input to extract streams in case if you wanted another person to extract their very own streams using your own findings and patch, well patch compares two inputs and generates a diff file which can allow you to patch an input to make it similar to the new data.

Find

xtool find [parameters] extracted_streams original_data [decode_data]

Find can be used to search for streams to then produce decode data which you can upload here on the forum in cases where you wanted to inform people what files should be extracted if they wanted for example to separate audio files per language from a game.

An example where I wanted to make a guide of how to separate english, german, italian, russian and spanish languages from the game would be something like:

xtool find "extracted\en\*" "game\*" en.xtl
xtool find "extracted\de\*" "game\*" de.xtl
xtool find "extracted\it\*" "game\*" it.xtl
xtool find "extracted\es\*" "game\*" es.xtl
xtool find "extracted\ru\*" "game\*" ru.xtl

These files are then uploaded for people if they wanted to extract the same data by themselves by using xtool via the "extract" function or for later use by yourself.

Extract

xtool extract decode_data original_data extracted_streams

Extract works mostly hand in hand with with the find function as its job is to use the decode_data to extract the data that was used for its generation.

A use case for it would be similar to the find example but from another user's perspective, so if someone uploaded decode data to use to extract my very own streams from their investigations, rather than them giving me position ranges for languages or the tools to use, we can just use their uploaded decode data to do it ourselves

xtool extract en.xtl "game\*" "extracted\en\*"
xtool extract de.xtl "game\*" "extracted\de\*"
xtool extract it.xtl "game\*" "extracted\it\*"
xtool extract es.xtl "game\*" "extracted\es\*"
xtool extract ru.xtl "game\*" "extracted\ru\*"

Then from here, we use these extracted files, to even use the erase function introduced in the version prior to blank out the sectors to prepare our own repack

xtool erase "extracted\*" "game\*" languages.xtl

which we can then restore after installation using

xtool decode languages.xtl "extracted\*" "game\*"

Patch

xtool patch [parameters] old_data new_data patch_data
xtool decode patch_data old_data

Patch's use case can be the generation of update patches in cases where you already made a repack but for whatever reason refuse to re-repack the game so instead choose to generate a patch file which can be used to update the old repack with newer files

xtool patch "Sims4\*" "Sims4_updated\*" sims4_wedding_stories.patch

Then after the installation of the original repack, you can add an additional archive that has sims4_wedding_stories.patch which will be used to update the game

xtool decode sims4_wedding_stories.patch "Sims4\*"

I noticed after compiling xtool.exe that xdelta is set to compress these patch files which means that they cannot be precompressed nor compressed with something better, however this compression in the next version will be removed. :)

[Razor12911]

Update available

Changes

- added IO functions (archive, execute)
- fixed issue in patch io function
- removed compression on patch diff files

Notes

More IO functions added. Archive behaves like -m0 but allows you to the archive to stdout and read from stdin when decoding (if you ever need that). Then there is execute which allows you execute several instances of another executable in parallel mode while all their inputs are fed from one source and all their output is fed to one destination.

Archive

xtool archive files1 files2... archive

There is nothing to add here as I have personal uses for this but the example would be

xtool archive game\* mygame.xtl
xtool decode mygame.xtl extracted\*

Execute

xtool execute [parameters] input output [exec_syntax]

Too lazy to write description but, here's an example

xtool.exe execute -c64mb -t8 UI.sb UI.bin bcm.exe -b64 [filein] [fileout]
xtool.exe decode -t100p UI.bin UI_dec.sb bcm.exe -d [filein] [fileout]

The left side of the syntax is to command xtool and after specifying input and output files, the right side begins and here you can write the command line that should be used to perform execution.

[filein], [fileout], [stdin], [stdout] can be used and denote what IO the program being executed uses.
[] is used to avoid conflicting with Freearc's <>

Freearc example would look like this

[External compressor:xbcm]
header = 0
packcmd = xtool.exe execute { -option} - - <stdin> <stdout> bcm.exe -b64 [filein] [fileout]
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout> bcm.exe -d [filein] [fileout]

and the method would be -mxbcm:c64mb:t75p

[Razor12911]

Update available

Changes

- added png stream preprocessor
- removed grittibanzli codec (since nobody uses it)

Notes

Xtool is able to process deflate/zlib streams and png images do contain these streams however, they are split up into several blocks which at times does prevent the program from being able to process them. The png stream preprocessor's job is to concatenate (rejoin all these blocks into a single stream) which can then be processed by xtool, so if you know your input contains these images then it's best to include -mpng into the method chain and use -d1 to first preprocess the streams then process them using zlib, reflate or preflate (preflate is the preferred method to use).

Results

without png preprocessor:
Compressed 1 file, 7,232,549 => 8,291,632 bytes. Ratio 114.64% -mreflate
Compressed 1 file, 7,232,549 => 7,232,655 bytes. Ratio 100.00% -mpreflate

with png preprocessor:
Compressed 1 file, 7,232,549 => 26,075,119 bytes. Ratio 360.52% -mpng+reflate -d1
Compressed 1 file, 7,232,549 => 25,289,529 bytes. Ratio 349.66% -mpng+preflate -d1

[Cesar82]

@Razor12911, can i use the parameters in that order in freearc commandline?
xtool:mpreflate:mpng:d1

[Razor12911]

yes but you'll have to change arc.ini from { -moption} to { -option}

[Cesar82]

yes but you'll have to change arc.ini from { -moption} to { -option}

Thanks.
Yes, I use { -option} in Arc.ini of the DSG.
I tested precompressing pngs with just "xtool:mpreflate" or with "xtool:mpreflate:mpng:d1" and got similar (near)I figured I was using the command order wrong.

[yasitha]

sorry, for asking this question here, i don't know where to ask.

someone please tell me and give me,
best .Wav (.pck) compressor
with arc.ini info and required files.
Thanks 😊

hint:
i try Pck files, on GFC.
Game File Scanner showing WAV streams. so i guess i can use Wav compressor.

[KaktoR]

msc :confused:

And I don't know what this has to do with xtool.

[yasitha]

msc :confused:

And I don't know what this has to do with xtool.

yes, my bad, this is not related to xtool
i try msc, but wont work. can you help me. :)

[kj911]

yasitha: Try it use testing tta methods from FreeArc and totally ignored file extension from during compression.

This compression oks, use final packing the following switches:

tta+srep+lzma or lolz

[shazzla]

Afaik freearc's tta works on per file basis,while msc works on stream(/file)?!