Regarding the crilayla algo - it's speed heavily depends on data and data size. Big files with RAW-like data compress insanely slowly, even with original algo (not xtool). So when xtool+cri is slow - it's not because of xtool, but because of crilayla. And with big files/chunks sometimes only one thread works in xtool at any given second and doesn't start next chunks until the big one is finished.

^Yes that is to be expected as threads have to wait for each other at some point to advance It should still be faster than one by one step of bms tool, although using your style where you dump raw files and use ppx to parallel them is of course even better.
But I think in those Persona 5 packs some files are not compressed(in the same file). They(some) don't have "CRILAYLA" header and dealing with all that really add to work. I could just dump ones with header of course as those without are likely uncompresable like audio and videos anyway.
But with xtool this saves so much hassle for just little extra time.


EDIT:I tried another cpk file with FA GUI and c32mb, I am getting about ~9mb/s inflating and ~10mb/s recoding it. ~10mb/s is IMO more than ok and I saw someone's oodle giving only slightly higher speed. Deflate gives me about 20mb/s(decompression) so 10mb/s for crilayla is great.

Regarding '+' and ',' characters issues, why not just use ':' for everything and dissect the parameters in the code instead? I mean, for example:
^This is perfectly easy to scan because names like zlib, kraken, reflate etc are unique and cannot appear as codec flags. So in your code you would just look for a word that represent a codec in the list and then all other params would be flags until next codec name is detected. So for example:
program would still know that zlib is with default parameters and kraken with whatever you put behind it simply because it knows about reserved words.

Also regarding versioning, whats wrong with simple v1.032, v1.033, v1.034 or anything like that?

Just a suggestions of course, I don't want to feel ungrateful or anything.

Sorry for spamming but another zip file test:
3.4% better final compression of grittibanzli vs reflate. Preflate look weak so far in every test for me. Looks like zlib+reflate and/or grittibanzli are 2 best general options.

EDIT:tried on another 2g file, grittibanzli took a freaking 20(!) minutes and loaded all cpu cores to max nonstop, pzlib+deflate took ~2min with much smaller cpu activity. Final compression was around 860mb vs 844mb only. That leave pzlib+deflate the only and best viable method for me.

Yes, zlib is supposed to handle deflate but here's the catch, most of the standard zip files does not use regular deflate levels. In order to precompress, the zlib codec tries standard deflate compression levels (9levels * 9 memlevels = 81 tries) and not the tuned deflate options. On the other hand, preflate and reflate does not go with the hit&trial method, it has it's own algo for processing deflate streams. Games in general does not use deflate tuning which is why the zlib codec works good for games, but in other cases like zip, preflate/reflate works better.

Using zlib on where it does not need preflate/reflate is better though, as zlib codec does not produce additional hif files which makes compression a bit worse. When you are not sure, zlib+reflate is good to go as zlib will precompress what it can and pass whatever it can't to reflate which will maximize precompression. Except a few complex deflate streams, zlib+preflate is also a good option as hif files produced by preflate are significantly smaller than the files produced by reflate.

Cyberpunk 2077 (Preload)

Test on basegame_1_engine.archive

xtool2020 (mkraken)

pack

unpack

@dixen
Which oodle core used? One from game file?

Update available

Changes

- added oo2ext* dll support
- updated search support

To everyone

If you left a post and I didn't reply to or acknowledge, I'm a bit busy and I spend way less time on forum.

Regarding the project. I don't plan on making a 2021 release of xtool, the 2020 is more of a final release and it will only receive minor updates and bug fixes with no new features. If there is a game that requires support, you need to check the plugins that are designed for xtool. I'll be focusing more on this. For the next few days I shall be giving out updates until Christmas so if there is a major feature that you need to be added in xtool leave a post and I'll probably add it.

I do recall having a few issues with new XTool whereit would stop during decompression of ue4 games.

I will do a test and see if that's still an issue.

Mr. Razor
Is lz4, lzo and zstd still not supported for xtool 2020?

Razor, pls check my above post No. 243 and consider that idea for parameters parsing to finalize specification. It would make xtool compatible with FA including GUI.
Finally, please consider including all parameters list with their description within command line help instead of external file.

EDIT: Also, provide full documentation on plugin system reserved keywords and their exact meaning(like if offset jump is meant from current position or beginning, if reading string advance position or not and so on), their full list etc.

Thank you for everything.

@Razor12911, is there any change from XTool 2020 to have a 32 bit version too?
Perhaps in the thin version l of the XTool 2020, it would be great.
Most games require 64-bit systems, but there are still some games that support 32-bit.
So if there is only 64-bit XTool it would not be recommended to compress this type of games using it, having to choose to use the old XTool.
Also if the user has a 32-bit system installed he will not be able to use Xtool, even if it is for a game that will be installed on another system in the future.
Thanks!

Why? What would happen, according to you? 64/32bit xtool have absolutely no relation to 64/32bit games. Only libraries used in xtool, hence why crilayla was dropped.

I think people with 32-bit operating systems use 32-bit versions of games. If they want to back up their games, they must use the 32-bit compression tool. Yes, xtool's architecture (32 or 64) is for the operating system only and not for the game.

f you create a compressed Data.bin file using 64-bit XTool we will not have to extract (install) on 32-bit systems.

The question was addressed to Razor12911. Most of the time other users are commenting and giving their answers and it turns out that the question post is left behind and the user who was asked many times ends up not seeing the question.
Please, other users, avoid answering questions that have a username mentioned, so it is easier to be answered. Thanks.

"File not found"

btw, remember how this xtool works. Everything is decided by the user, before when you used zlib in the older xtool and ztool, if reflate library were nearby, it would use them so perhaps you need to combine zlib and reflate to get the desired output.

All issues related to libdunia will be fixed via a separate plugin of xtool

I temporarily removed crilayla because I wanted to have a reworked 64-bit version via plugin support, not many games use it so it shouldn't ship with xtool. Oodle, zlib, lz4 and etc, these algorithms are common therefore they ship with xtool, anything else will be a plugin.

zlib is the first codec that tries to process streams, if you mixed the methods zlib+reflate... if zlib fails, it gives it to reflate to try processing it.

lz4 and zstd are in xtool but lz4 is just a placeholder for now, lzo will be added as a placeholder too.

regarding character parsing, I could change it but the thing is, xtool doesn't really know reserved words because there are none but I can change it if that's what people want.

The reason I went with a versioning like this is because the there are already 3 different xtools so to not confuse people, they should just check the YYMM it was released on as the old xtool, 0.0.19... has special editions from the request of repackers so basically the versioning is so confusing so I then decided I'd only adopt it once I think the project is a guaranteed success. 2019 is an example because what I tried to do fail, so I tried again with 2020 to see if it will work with little hope of success but as soon as it seems like the idea of external plugins is possible and can be done without bugs then versions will be brought back.

I don't know if I said it here on the forum or to someone who asked for 32-bit version of xtool. I stopped shipping 32-bit xtool because there are certain mistakes that some people make when they use xtool. An example is when they use oodle/lz4. The library you use must be compiled from the same source code, this you must make sure (if the 64-bit dll of lz4 is v191, make sure the 32-bit dll is also v191). So what happens is people use 64-bit version of the library and then ship the 32-bit version of xtool along with different libraries making xtool to cause CRC errors because the libraries didn't come from the same source. 32-bit is available on request but it will be the users fault if they have issues with it. What you also must remember is xtool ships with xdelta, some streams cannot be restored so it tries to salvage a few streams but if there is a mismatch in the data caused by different dll, it causes xtool/xdelta to return errors.

Relax I backtrack posts when I return and answer all of them when I have time, I did say that I spend less time here and sometimes what happens is if a question is directed at someone, if they are not around and other people don't step in, it still falls behind and the question is still never seen (especially with threads). So I think he did what was best by giving a response right away best he can according to his knowledge.

Thanks for the quick response.

If I understand correctly, users compress a game with a library version and try to extract it with another.
But can't this happen with only the 64-bit version of XTool (Compressing with XTool X and trying to extract with XTool version Y)?
You always share the libraries together with XTool, so you should logically use them to decompress.

My idea was to update XTool on CIU so I decided to ask.
Now in CIU the libraries and decompressors are no longer included in the setup.exe, being added when the game is deleted in an external file, thus avoiding a bit of using different versions of the DLL and exes for extraction.

You said always go back on the posts and respond.
You ciu this question referring to ue4dt in the post HERE.

Can I have specific plug-in files for a game like "oo2ext_7_win64.dll / cp2077.ini" inside the XTool folder when using other methods?
If I want to include several plugins (specific files needed for a particular game) I can put them all in a folder with XTool and in the files for decompression do not include these files, or they must also be together in the extraction, even if it is specific to another method ? Of course, if the libraries in the "Libraries" folder shared with XTool exist alongside XTool in compression, must be together in decompression.

No you don't understand. The libraries have versions of their own. What I am saying is, if you used 64-bit version of xtool.exe and liblz4.dll when encoding. Repackers and other users have that thing of putting xtool.exe (x86) this is not a problem since xtool x86 and x64 are usually compatible with each other, the problem is then the liblz4.dll, users just look for any x86 liblz4.dll just because they want it to work with xtool.exe x86 even when liblz4.dll x86 and liblz4.dll are not the same version. Almost each version of liblz4.dll produces a different result when its used which causes crc errors for xtool.

This is user error, someone tried to help you solve this issue in the next post.
https://www.fileforums.com/showpost....2&postcount=77
therefore I didn't intervene.
-mue4dt:-m1:-k0x115EE4F8C625C792F37A503308048E79726E512F0BF8D2A D7C4C87BC5947CBA7+xzlib+srep:m3f+4x4:lzma+diskspan :4420mb:4470mb
the problem here is simple, you added -mue4dt:-m1:-k0x...

I don't know if that was the issue or with the program itself.

not sure if I follow but you can probably do that, add several plugins. Even if they are not inuse. Or tell users to copy and paste the plugin info (ini) to a static file that you will use.
For example you can just make a custom configuration file, like extmethod.ini in your tools or CIU.
If a user wanted to use a certain plugin, instead of copying cp2077.ini, they can just copy its contents to extmethod.ini. this is also allowed. You can put in as many methods in there as you want but they must follow the standard ini section. [Stream1] ... [Stream2]... and etc. These configuration files as I mentioned are not required for decoding.

Remember, placing oo2ext_7_win64.dll as an example. This means xtool will use that library for everything oodle related so if your question was, can you make a folder which xtool will read cp2077 where there will be (oo2ext_7_win64.dll and cp2077.ini) and another game tc2 which uses (oo2ext_5_win64.dll and tc2.ini), xtool will use oo2ext_5_win64.dll for all of them.

But you usually share the libraries with XTool.
So if any user is going to replace Xtool with XTool x86 they must also replace the libraries with the libraries that come with XTool x86 (In the file you shared from xtool x86).
I cannot include "-" in the parameter.
So that must be the mistake. I'll check it out and send it to KaktoR to test it, but I think that's what happened. Thanks!
I did not pay attention to the user's Masquerade response. If he had highlighted the "-" he would have noticed.
It's even generating the wrong method on the DiskSpan_GUI that I created...
https://i.imgur.com/1hJDlir.png

So the ideal would be to put just XTool and the common libraries, and put the plugins in separate folders and create copies of these plugins to the XTool folder when initializing the compression and deleting after completion.

Perhaps a configuration file with a path for additional plugins would be a idea.
Example: Use an XTool.ini file and place keys such as Path1=Plugin1, Path2=Plugin2, etc.
When starting XTool, not only load libraries and files that are in the XTool path, but also check and load the subfolders with those paths.
The scan of the keys would have to be checked as long as the key Path#= exists, and not if it is empty because to disable a plugin it would only be necessary to pass an empty value to key path#=. This is only an idea if it doesn't hinder the performance of XTool.

Razor12911, Thank you for the information.
It's very important to know because I've faced this situation before. It took a long time to find the cause of the CRC error. But I failed.

Razor12911, seems like you broke xtool:kraken with latest version.
I only tested on Death Stranding (of course, decrypted with dst)...you can easily test it yourself with your own dst_r1 package.
Use your original pack.bat -> data.arc is inflated (using oo2reck).
Use xtool_2011_r1 xtool.exe with xtool:kraken -> data.arc is inflated.
Use xtool_2012_r1 xtool.exe with xtool:kraken -> data.arc is NOT inflated.

P.S. Why am I using xtool for this when everybody is using oo2reck? Because xtool:kraken decompress FASTER than oo2reck. I don't care if compression is longer (and it is), decompression speed is more important for me.

I guess I have the same. kraken does not work with latest version (and I thought I was just to dumb :p).

Is there a progress update on Watch Dogs precompressor, or if not, some guidance on how to write an xtool config in order to create one?

Update available

Changes

- added library support
- added compress, decompress, encrypt, decrypt, hash, delta functions (used by library)
- added lzo codec placeholders
- fixed oodle bug
- fixed lz4 bug
- removed libdunia codec

@infovs, KaktoR
Should be fixed now

@Masquerade
Wait for documentation update

To everyone

This is the final major update of xtool, this is a tiresome project and from now onwards it will only be getting minor updates and bug fixes. If you're worried about support for games released in future which xtool has no support of, I have added 3 ways of adding plugins to xtool and that's either by a database, a configuration or a plugin that is written in Delphi or C++ (a well written documentation that explains how these work is underway).

I removed libdunia because this codec is not suppose to be part of xtool, I added it become some scene groups were on the roll in bypassing Denuvo so I thought they have this one in the bag but turns out people have to wait a little longer which gave me time to finish library support. Library support is basically dlls written to add additional support to xtool without updating the main project. Think of it like the CLS of Xtool so be sure to check Plugins thread for the separate plugin.

__________________

I wish everyone a Merry Christmas and a Happy New Year.
This year was terrible and I have high hopes for the next one so see you then. :)

oodle is fixed, thanks :) Merry christmas to you too!

I am inflating Cyberpunk 2077 with cp2077 plugin. All *.archive tarred through groups in FA. I am at 32gb in tmp file right now and total of all archives is 60gb so I am still only half the size of original compressed data. This is after 6h10min. In FA gui it says speed is at 0.11mb/s. Is this supposed to be this slow? -t100p and 4 core 4690k 4.2ghz

PS(first ~27gb or so went much quicker than is now, but it is working and did not hanged)

@elit: No it's not supposed to be that long.

Exactly! But.. I doubt I am doing anything wrong, I mean:
*.archive = $gamelz group.
So *.archive = FA -mxtool:mcp2077+srp64+xlzma:lc8
or xtool precomp -t100p -mcp2077 to be precise
Anyone?(PS its latest 2012r2)
EDIT(restarted without t100p, but I don't think that's it: Attachment 28650
How you guys getting ~10mb/s is beyond me, are you tarring or doing individual files? As it may matter..)
EDIT2(speed down to 0.6mb/s, estimation ~22h+ and slowing down. Really, this ain't right but tool appear to be working correct not hanging! Can anyone try: latest xtool version with cp2077 plugin and on tarred archives through FA not individual files? I am sure there is something wrong in one of those variables.)
PS(I forgot to note that game is gog with 1.06 update, that means 9gb 1.05 was also applied, maybe it changed oodle structure? I will postpone compression until solution/cause is clear.)

So I just quickly tried oo2rect. That one seem to be working ok! I am going to use it on whole game and will see. I also tried xtool with -mkraken instead cp2077 plugin and also on individual archives. Speed issue remains. It does seem to have problem no matter what, while oo2rect seems fine.

EDIT:
oo2rect is same slow, after ~1h about 20gb and 0.xxmb/s speed. At this point I am pretty sure problem is not xtool nor oo2rect or plugin, but a game. If this was not a case with original version 1.03, try to update to 1.06, I think they changed data enough to become problem. For reference, files of concern are:
basegame_3_nightcity.archive
basegame_3_nightcity_gi.archive

Alright so I left it overnight. I can confirm it to work correctly but it took 11h:40min to compress, from that about ~10h due to xtool oodle. Archive tested ok and decompression was much quicker, compression is that slow though.

No Idea what's wrong, but for me it's working just fine.

I use GOG version 1.06, along with an Ryzen 5 2600 and 16GB RAM and latest xtool version posted a few days ago.

I can test the whole game folder if you wish, but I can tell you the results will be just fine and nothing wrong.

Xtool settings I use:

Hm, you have 6 core with hyperthreading. Maybe -t100p for you utilize all 12 virtual cores. You may try -t4 and also no --dbase to see if it affect speed that much. Also no need to test all archives only 2 files I mentioned above is enough. I wonder if --dbase or 12 virtual cores make for such difference. BTW thanks for the info above, it's interesting to know.

i5 4590, 4 Cores (4 Threads)

My concern is the difference in ratio, --dbase should not have an impact on ratio just the speed. Some bug I guess...

Here's an interesting one.

ZZ1.dat - 175MB from Steel Division 2.


XTool 2020 (zlib)
Xtool v12 (zlib)
XTool v12 hangs inifintely when during decompression - the file just keeps getting bigger and bigger.

GFS Detects zlib streams, whereas Drop Scan does not:

https://i.imgur.com/YHG8hpl.png

https://i.imgur.com/6n0GrEA.png

https://anonfiles.com/FcY1Q060p9/ZZ_1_dat

Make headerless+force detection

PS: The anonfiles is shit. Always slow connection here :D (185kb/s)

@Masquerade
Use reflate codec

When you use zlib method in Xtool v12 or older variants of xtool, they automatically used reflate when you placed the libraries each time zlib fails, the new xtool does not do any of this. The user is the one who needs to do this by themselves because what I have noticed is people putting reflate libraries near xtool and they never know when xtool uses them or when it doesn't. This way the user knows what the issues is when decompression fails each time when they try to figure out what the actual problem is.

I also did point out that the best method for precompressing anything that is zlib/deflate compressed, use -mzlib along with either reflate/preflate so you get the best results.