XTool 2020 (Main Project)

[Cesar82]

Originally Posted by elit

Why? What would happen, according to you? 64/32bit xtool have absolutely no relation to 64/32bit games. Only libraries used in xtool, hence why crilayla was dropped.

f you create a compressed Data.bin file using 64-bit XTool we will not have to extract (install) on 32-bit systems.

The question was addressed to Razor12911. Most of the time other users are commenting and giving their answers and it turns out that the question post is left behind and the user who was asked many times ends up not seeing the question.
Please, other users, avoid answering questions that have a username mentioned, so it is easier to be answered. Thanks.

[Razor12911]

Originally Posted by elit

You may want to be interested in this.

I was trying to inflate cpk file from Diablo 3 from nintendo switch. Normaly these are crilayla but gfs detected lz. So I tried old ztool, some older xtool(date modified says 26.april 2019) and xtool2009R3.

For reference I tried single file "Act1.cpk" of 814mb size. GFS detected potential of inflation of up to ~1800mb.

Ztool could not inflate past few 10's of mb if any, regardless of setting like m2, m3 etc.

Later xtool2009R3 depended on codec, most were at about 100mb extra only, except reflate which did go to 1.1gb. However final packed size after srep+xlzma was about same which make me think of possible huge overheat?

Finally, that older xtool with :high setting was able to inflate to.. 1.7gb!!
I tried to compress that and got about ~40mb better compression.

General final compression after codecs was around 740mb+, this one went to ~704mb. I wonder if this was real inflated data to 1.7gb or too much overheat. I must also say that older xtool was able to do it only with :high option which unlike ztool is not even know or documented and probably unofficial. Also it took very long time compare to say ztool:high, it seems to work differently.

I have uploaded the file act1.cpk for further research if you are interested:
https://megaup.net/1zEg1/Act1.cpk

"File not found"

btw, remember how this xtool works. Everything is decided by the user, before when you used zlib in the older xtool and ztool, if reflate library were nearby, it would use them so perhaps you need to combine zlib and reflate to get the desired output.

Originally Posted by Masquerade

Having a few problems decompressing with the fcp option, I have tried a few different unpackcmds:

[External compressor: fcp]
header = 0
unpackcmd = xtool decode --dbase=fcp.xtl - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode - - <stdin> <stdout>

[External compressor: fcp]
header = 0
unpackcmd = xtool decode -mfcp - - <stdin> <stdout>

Yes, fcp.xtl and liblz4.dll are with xtool.exe

Errors like:

G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: general (de)compression error in fcp


G:\Games\_Compressor>aarc x __test.msq
FreeArc 0.67 (March 15 2014) extracting archive: __test.msq
Extracting 1 file, 46,201,655 bytes. Processed   0%
ERROR: write error (disk full?) in compression algorithm fcp

This is just on 1 file with no extra compressors.

Is there something wrong with my unpackcmd?

All issues related to libdunia will be fixed via a separate plugin of xtool

Originally Posted by elit

In fact you already completed the work, why abandon it? So for the love of god I beg you, put it back in latest version!

PS(Zstd and lz4 are temporary and will stop working after time because of their retarded design. Not sure about lzo(that is actually a question, why is standard lzo not working in dunia and unreal engines and need special treatment? I thought its a stable design like zlib?). But crilayla, lzss, yaz0 and such should remain compatible.

I temporarily removed crilayla because I wanted to have a reworked 64-bit version via plugin support, not many games use it so it shouldn't ship with xtool. Oodle, zlib, lz4 and etc, these algorithms are common therefore they ship with xtool, anything else will be a plugin.

Originally Posted by elit

-grittibanzli inflate completely randomly. Sometimes tested 20mb inflated zip file is 34mb, other times 31.6mb, another time 33mb atd. exactly same setting and just repeating same command on console.

-zlib codec did not inflated zip file at all!(I thought it was supposed to handle deflate?)

-preflate and reflate both did to same ~27mb, but preflate took ~6sec while reflate did it in ~1sec.

Looks like good old reflate is still the best zlib codec to stick.


EDIT(and ztool is also able to inflate to 27mb without :high or :m2 !) <<UPDATE(BS, scrap that claim, need m2 or high indeed)
EDIT2(ok I checked manual again and learned about zlib+reflate combination benefits. Maybe this is best way. It took ~3sec compared to ~1sec on deflate alone though. Not sure how big those benefits are in real application yet.)
EDIT3(have to explicitly state 'mb' in -c option instead of just 'm', e.g. -c128m vs -c128mb. No big deal but used to ztool, was more flexible. PS: I am really liking the idea of zlib+reflate in one go, before I used to pass twice in test to find out which one to use.)

zlib is the first codec that tries to process streams, if you mixed the methods zlib+reflate... if zlib fails, it gives it to reflate to try processing it.

Originally Posted by Ele

Mr. Razor
Is lz4, lzo and zstd still not supported for xtool 2020?

lz4 and zstd are in xtool but lz4 is just a placeholder for now, lzo will be added as a placeholder too.

[Razor12911]

Originally Posted by elit

Regarding '+' and ',' characters issues, why not just use ':' for everything and dissect the parameters in the code instead? I mean, for example:

xtool -mzlib:c32mb:reflate:l9

^This is perfectly easy to scan because names like zlib, kraken, reflate etc are unique and cannot appear as codec flags. So in your code you would just look for a word that represent a codec in the list and then all other params would be flags until next codec name is detected. So for example:

-mzlib:kraken:{param}

program would still know that zlib is with default parameters and kraken with whatever you put behind it simply because it knows about reserved words.

Also regarding versioning, whats wrong with simple v1.032, v1.033, v1.034 or anything like that?

Just a suggestions of course, I don't want to feel ungrateful or anything.

regarding character parsing, I could change it but the thing is, xtool doesn't really know reserved words because there are none but I can change it if that's what people want.

The reason I went with a versioning like this is because the there are already 3 different xtools so to not confuse people, they should just check the YYMM it was released on as the old xtool, 0.0.19... has special editions from the request of repackers so basically the versioning is so confusing so I then decided I'd only adopt it once I think the project is a guaranteed success. 2019 is an example because what I tried to do fail, so I tried again with 2020 to see if it will work with little hope of success but as soon as it seems like the idea of external plugins is possible and can be done without bugs then versions will be brought back.

[Razor12911]

Originally Posted by Cesar82

@Razor12911, is there any change from XTool 2020 to have a 32 bit version too?
Perhaps in the thin version l of the XTool 2020, it would be great.
Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it, having to choose to use the old XTool.
Also if the user has a 32-bit system installed he will not be able to use Xtool, even if it is for a game that will be installed on another system in the future.
Thanks!

I don't know if I said it here on the forum or to someone who asked for 32-bit version of xtool. I stopped shipping 32-bit xtool because there are certain mistakes that some people make when they use xtool. An example is when they use oodle/lz4. The library you use must be compiled from the same source code, this you must make sure (if the 64-bit dll of lz4 is v191, make sure the 32-bit dll is also v191). So what happens is people use 64-bit version of the library and then ship the 32-bit version of xtool along with different libraries making xtool to cause CRC errors because the libraries didn't come from the same source. 32-bit is available on request but it will be the users fault if they have issues with it. What you also must remember is xtool ships with xdelta, some streams cannot be restored so it tries to salvage a few streams but if there is a mismatch in the data caused by different dll, it causes xtool/xdelta to return errors.

Originally Posted by Cesar82

The question was addressed to Razor12911. Most of the time other users are commenting and giving their answers and it turns out that the question post is left behind and the user who was asked many times ends up not seeing the question.
Please, other users, avoid answering questions that have a username mentioned, so it is easier to be answered. Thanks.

Relax I backtrack posts when I return and answer all of them when I have time, I did say that I spend less time here and sometimes what happens is if a question is directed at someone, if they are not around and other people don't step in, it still falls behind and the question is still never seen (especially with threads). So I think he did what was best by giving a response right away best he can according to his knowledge.

[Cesar82]

Originally Posted by Razor12911

I don't know if I said it here on the forum or to someone who asked for 32-bit version of xtool. I stopped shipping 32-bit xtool because there are certain mistakes that some people make when they use xtool. An example is when they use oodle/lz4. The library you use must be compiled from the same source code, this you must make sure (if the 64-bit dll of lz4 is v191, make sure the 32-bit dll is also v191). So what happens is people use 64-bit version of the library and then ship the 32-bit version of xtool along with different libraries making xtool to cause CRC errors because the libraries didn't come from the same source. 32-bit is available on request but it will be the users fault if they have issues with it. What you also must remember is xtool ships with xdelta, some streams cannot be restored so it tries to salvage a few streams but if there is a mismatch in the data caused by different dll, it causes xtool/xdelta to return errors.



Relax I backtrack posts when I return and answer all of them when I have time, I did say that I spend less time here and sometimes what happens is if a question is directed at someone, if they are not around and other people don't step in, it still falls behind and the question is still never seen (especially with threads). So I think he did what was best by giving a response right away best he can according to his knowledge.

Thanks for the quick response.

If I understand correctly, users compress a game with a library version and try to extract it with another.
But can't this happen with only the 64-bit version of XTool (Compressing with XTool X and trying to extract with XTool version Y)?
You always share the libraries together with XTool, so you should logically use them to decompress.

My idea was to update XTool on CIU so I decided to ask.
Now in CIU the libraries and decompressors are no longer included in the setup.exe, being added when the game is deleted in an external file, thus avoiding a bit of using different versions of the DLL and exes for extraction.

You said always go back on the posts and respond.
You ciu this question referring to ue4dt in the post HERE.

Can I have specific plug-in files for a game like "oo2ext_7_win64.dll / cp2077.ini" inside the XTool folder when using other methods?
If I want to include several plugins (specific files needed for a particular game) I can put them all in a folder with XTool and in the files for decompression do not include these files, or they must also be together in the extraction, even if it is specific to another method ? Of course, if the libraries in the "Libraries" folder shared with XTool exist alongside XTool in compression, must be together in decompression.

[Razor12911]

Originally Posted by Cesar82

Thanks for the quick response.

If I understand correctly, users compress a game with a library version and try to extract it with another.
But can't this happen with only the 64-bit version of XTool (Compressing with XTool X and trying to extract with XTool version Y)?
You always share the libraries together with XTool, so you should logically use them to decompress.

No you don't understand. The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.

You said always go back on the posts and respond.
You ciu this question referring to ue4dt in the post HERE.

This is user error, someone tried to help you solve this issue in the next post.
https://www.fileforums.com/showpost....2&postcount=77
therefore I didn't intervene.
-mue4dt:-m1:-k0x115EE4F8C625C792F37A503308048E79726E512F0BF8D2A D7C4C87BC5947CBA7+xzlib+srep:m3f+4x4:lzma+diskspan :4420mb:4470mb
the problem here is simple, you added -mue4dt:-m1:-k0x...

I don't know if that was the issue or with the program itself.

Can I have specific plug-in files for a game like "oo2ext_7_win64.dll / cp2077.ini" inside the XTool folder when using other methods?
If I want to include several plugins (specific files needed for a particular game) I can put them all in a folder with XTool and in the files for decompression do not include these files, or they must also be together in the extraction, even if it is specific to another method ? Of course, if the libraries in the "Libraries" folder shared with XTool exist alongside XTool in compression, must be together in decompression.

not sure if I follow but you can probably do that, add several plugins. Even if they are not inuse. Or tell users to copy and paste the plugin info (ini) to a static file that you will use.
For example you can just make a custom configuration file, like extmethod.ini in your tools or CIU.
If a user wanted to use a certain plugin, instead of copying cp2077.ini, they can just copy its contents to extmethod.ini. this is also allowed. You can put in as many methods in there as you want but they must follow the standard ini section. [Stream1] ... [Stream2]... and etc. These configuration files as I mentioned are not required for decoding.

Remember, placing oo2ext_7_win64.dll as an example. This means xtool will use that library for everything oodle related so if your question was, can you make a folder which xtool will read cp2077 where there will be (oo2ext_7_win64.dll and cp2077.ini) and another game tc2 which uses (oo2ext_5_win64.dll and tc2.ini), xtool will use oo2ext_5_win64.dll for all of them.

[Cesar82]

Originally Posted by Razor12911

No you don't understand. The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.

But you usually share the libraries with XTool.
So if any user is going to replace Xtool with XTool x86 they must also replace the libraries with the libraries that come with XTool x86 (In the file you shared from xtool x86).

Originally Posted by Razor12911

the problem here is simple, you added -mue4dt:-m1:-k0x...

I cannot include "-" in the parameter.
So that must be the mistake. I'll check it out and send it to KaktoR to test it, but I think that's what happened. Thanks!
I did not pay attention to the user's Masquerade response. If he had highlighted the "-" he would have noticed.
It's even generating the wrong method on the DiskSpan_GUI that I created...

Originally Posted by Razor12911

not sure if I follow but you can probably do that, add several plugins. Even if they are not inuse. Or tell users to copy and paste the plugin info (ini) to a static file that you will use.
For example you can just make a custom configuration file, like extmethod.ini in your tools or CIU.
If a user wanted to use a certain plugin, instead of copying cp2077.ini, they can just copy its contents to extmethod.ini. this is also allowed. You can put in as many methods in there as you want but they must follow the standard ini section. [Stream1] ... [Stream2]... and etc. These configuration files as I mentioned are not required for decoding.

Remember, placing oo2ext_7_win64.dll as an example. This means xtool will use that library for everything oodle related so if your question was, can you make a folder which xtool will read cp2077 where there will be (oo2ext_7_win64.dll and cp2077.ini) and another game tc2 which uses (oo2ext_5_win64.dll and tc2.ini), xtool will use oo2ext_5_win64.dll for all of them.

So the ideal would be to put just XTool and the common libraries, and put the plugins in separate folders and create copies of these plugins to the XTool folder when initializing the compression and deleting after completion.

Perhaps a configuration file with a path for additional plugins would be a idea.
Example: Use an XTool.ini file and place keys such as Path1=Plugin1, Path2=Plugin2, etc.
When starting XTool, not only load libraries and files that are in the XTool path, but also check and load the subfolders with those paths.
The scan of the keys would have to be checked as long as the key Path#= exists, and not if it is empty because to disable a plugin it would only be necessary to pass an empty value to key path#=. This is only an idea if it doesn't hinder the performance of XTool.

[Ele]

Originally Posted by Razor12911

The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.

Razor12911, Thank you for the information.
It's very important to know because I've faced this situation before. It took a long time to find the cause of the CRC error. But I failed.

[infovs]

Razor12911, seems like you broke xtool:kraken with latest version.
I only tested on Death Stranding (of course, decrypted with dst)...you can easily test it yourself with your own dst_r1 package.
Use your original pack.bat -> data.arc is inflated (using oo2reck).
Use xtool_2011_r1 xtool.exe with xtool:kraken -> data.arc is inflated.
Use xtool_2012_r1 xtool.exe with xtool:kraken -> data.arc is NOT inflated.

P.S. Why am I using xtool for this when everybody is using oo2reck? Because xtool:kraken decompress FASTER than oo2reck. I don't care if compression is longer (and it is), decompression speed is more important for me.

[KaktoR]

I guess I have the same. kraken does not work with latest version (and I thought I was just to dumb ).

[Masquerade]

Is there a progress update on Watch Dogs precompressor, or if not, some guidance on how to write an xtool config in order to create one?

[Razor12911]

Update available

Changes

- added library support
- added compress, decompress, encrypt, decrypt, hash, delta functions (used by library)
- added lzo codec placeholders
- fixed oodle bug
- fixed lz4 bug
- removed libdunia codec

@infovs, KaktoR
Should be fixed now

@Masquerade
Wait for documentation update

To everyone

This is the final major update of xtool, this is a tiresome project and from now onwards it will only be getting minor updates and bug fixes. If you're worried about support for games released in future which xtool has no support of, I have added 3 ways of adding plugins to xtool and that's either by a database, a configuration or a plugin that is written in Delphi or C++ (a well written documentation that explains how these work is underway).

I removed libdunia because this codec is not suppose to be part of xtool, I added it become some scene groups were on the roll in bypassing Denuvo so I thought they have this one in the bag but turns out people have to wait a little longer which gave me time to finish library support. Library support is basically dlls written to add additional support to xtool without updating the main project. Think of it like the CLS of Xtool so be sure to check Plugins thread for the separate plugin.

__________________

I wish everyone a Merry Christmas and a Happy New Year.
This year was terrible and I have high hopes for the next one so see you then.

[KaktoR]

oodle is fixed, thanks Merry christmas to you too!

from rdr2

Compressed 1 file, 1,018,142,190 => 2,447,518,046 bytes. Ratio 240.39%
Compression time: cpu 0.88 sec/real 111.78 sec = 1%. Speed 9.11 mB/s
All OK

Extracted 1 file, 2,447,518,046 => 1,018,142,190 bytes. Ratio 240.39%
Extraction time: cpu 0.75 sec/real 91.56 sec = 1%. Speed 11.12 mB/s
All OK

[elit]

I am inflating Cyberpunk 2077 with cp2077 plugin. All *.archive tarred through groups in FA. I am at 32gb in tmp file right now and total of all archives is 60gb so I am still only half the size of original compressed data. This is after 6h10min. In FA gui it says speed is at 0.11mb/s. Is this supposed to be this slow? -t100p and 4 core 4690k 4.2ghz

PS(first ~27gb or so went much quicker than is now, but it is working and did not hanged)

[KaktoR]

@elit: No it's not supposed to be that long.

Overall input size: 59.64 GB
Overall output size: 125.20 GB
Overall conversion time: 01:23:28