XTool 2020 (Main Project)

[elit]

So I just quickly tried oo2rect. That one seem to be working ok! I am going to use it on whole game and will see. I also tried xtool with -mkraken instead cp2077 plugin and also on individual archives. Speed issue remains. It does seem to have problem no matter what, while oo2rect seems fine.

EDIT:
oo2rect is same slow, after ~1h about 20gb and 0.xxmb/s speed. At this point I am pretty sure problem is not xtool nor oo2rect or plugin, but a game. If this was not a case with original version 1.03, try to update to 1.06, I think they changed data enough to become problem. For reference, files of concern are:
basegame_3_nightcity.archive
basegame_3_nightcity_gi.archive

[elit]

Alright so I left it overnight. I can confirm it to work correctly but it took 11h:40min to compress, from that about ~10h due to xtool oodle. Archive tested ok and decompression was much quicker, compression is that slow though.

[KaktoR]

No Idea what's wrong, but for me it's working just fine.

I use GOG version 1.06, along with an Ryzen 5 2600 and 16GB RAM and latest xtool version posted a few days ago.

I can test the whole game folder if you wish, but I can tell you the results will be just fine and nothing wrong.

Xtool settings I use:

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mcp2077 -c128mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,714,137,974 bytes. Ratio 285.42%
Compression time: cpu 9.34 sec/real 704.44 sec = 1%. Speed 11.30 mB/s
All OK

Extracted 1 file, 22,714,137,974 => 7,958,224,896 bytes. Ratio 285.42%
Extraction time: cpu 8.11 sec/real 333.60 sec = 2%. Speed 23.86 mB/s
All OK

[elit]

Hm, you have 6 core with hyperthreading. Maybe -t100p for you utilize all 12 virtual cores. You may try -t4 and also no --dbase to see if it affect speed that much. Also no need to test all archives only 2 files I mentioned above is enough. I wonder if --dbase or 12 virtual cores make for such difference. BTW thanks for the info above, it's interesting to know.

[Razor12911]

i5 4590, 4 Cores (4 Threads)

[External compressor:xtool]
header    = 0
packcmd   = xtool.exe precomp -mcp2077 -c128mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,919,265,672 bytes. Ratio 287.99%
Compression time: cpu 9.91 sec/real 1538.09 sec = 1%. Speed 5.17 mB/s
All OK

Tested 1 file, 22,919,265,672 => 7,958,224,896 bytes. Ratio 287.99%
Testing time: cpu 6.45 sec/real 528.38 sec = 1%. Speed 15.06 mB/s
All OK

My concern is the difference in ratio, --dbase should not have an impact on ratio just the speed. Some bug I guess...

[Masquerade]

Here's an interesting one.

ZZ1.dat - 175MB from Steel Division 2.


XTool 2020 (zlib)

Compressed 1 file, 183,623,680 => 183,661,889 bytes. Ratio 100.02%
Compression time: cpu 0.16 sec/real 26.08 sec = 1%. Speed 7.04 mB/s
All OK

Xtool v12 (zlib)

Compressed 1 file, 183,623,680 => 226,689,420 bytes. Ratio 123.45%
Compression time: cpu 0.27 sec/real 26.18 sec = 1%. Speed 7.01 mB/s
All OK

XTool v12 hangs inifintely when during decompression - the file just keeps getting bigger and bigger.

GFS Detects zlib streams, whereas Drop Scan does not:





https://anonfiles.com/FcY1Q060p9/ZZ_1_dat

[KaktoR]

Make headerless+force detection

PS: The anonfiles is shit. Always slow connection here (185kb/s)

[KaktoR]

@Masquerade
Use reflate codec

Compressed 1 file, 183,623,680 => 227,188,297 bytes. Ratio 123.72%
Compression time: cpu 0.09 sec/real 5.12 sec = 2%. Speed 35.87 mB/s
All OK

Extracted 1 file, 227,188,297 => 183,623,680 bytes. Ratio 123.72%
Extraction time: cpu 0.13 sec/real 1.07 sec = 12%. Speed 171.41 mB/s
All OK

[Razor12911]

When you use zlib method in Xtool v12 or older variants of xtool, they automatically used reflate when you placed the libraries each time zlib fails, the new xtool does not do any of this. The user is the one who needs to do this by themselves because what I have noticed is people putting reflate libraries near xtool and they never know when xtool uses them or when it doesn't. This way the user knows what the issues is when decompression fails each time when they try to figure out what the actual problem is.

I also did point out that the best method for precompressing anything that is zlib/deflate compressed, use -mzlib along with either reflate/preflate so you get the best results.

[Masquerade]

Razor12911,
I did use zlib+preflate, I simply forgot about reflate.

After using relate, everything works fine.

[Razor12911]

Update available

Changes

- updated library support
- updated command line parser
- included x86 build
- fixed depthing issues

Notes

I have added 32-bit build as requested but you'll have to use x86 libraries or find a matching pair of x86/x64 libraries. You can get these from github of a project under the releases section.
The versioning is brought back to the format of 1.0.0 instead of the build number as requested.

[Masquerade]

@Razor12911

Thanks for this new method... incredible!

Compressing 1 file, 17,530,888,054 bytes
  Compressing TSCGame-WindowsNoEditor.pak
Compressed 1 file, 17,530,888,054 => 28,460,607,223 bytes. Ratio 162.35%
Compression time: cpu 18.42 sec/real 624.55 sec = 3%. Speed 28.07 mB/s
All OK

On the sample I sent to you yesterday.

[Cesar82]

@Razor12911, thanks for the 32-bit version, it will be very useful to include in CIU.
1) Also like to inform you that it is being detected as a false positive by the KIS anti-virus (I don't know if there is anything that can be done).
P.S: The 64-bit version has not been captured.

2) You could share libraries compatible with the 32-bit version to be fully compatible and avoid errors due to incompatibilities.
3) I wonder if I have 3 games (collection) and each game uses XTool with different methods to compress.
Assuming that only the libraries needed for compression (nothing more than necessary) for each Data.bin file are together with XTool.exe at the time of compressing.
To perform the decompression, can I have other libraries that were not used to compress the Data.bin file together with XTool (Like: Have 1 library next to XTool when compressing and 5 when decompressing)?
Thanks!

[Razor12911]

1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/...296d/detection
x64: https://www.virustotal.com/gui/file/...7dfe/detection

the code is actually the same so I don't know...

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
Capture2.PNG

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.

[Cesar82]

Originally Posted by Razor12911

1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/...296d/detection
x64: https://www.virustotal.com/gui/file/...7dfe/detection

the code is actually the same so I don't know...

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
Attachment 28717

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.

1) For me it displays "Item not found" message in the virustotal links of your post.
Thank you for the informations.
The false positive does not happen when using Windows Defender. I don't understand why KIS is capturing him. (Just because it's x86 (same source code)... It's like COVID, it reaches the weakest organisms (x86)).

2) I know that few use x86 systems. But because CIU still works on x86 systems it is useful to have x86 compatibility.
Maybe in some time, by default, in the CIU, the "ArchitecturesInstallIn64BitMode=x64" and "ArchitecturesAllowed=x64" configs would limit the CIU to only 64-bit systems so only 64-bit versions of compressors would be needed.
But for now, if possible, I prefer to maintain compatibility.
I know that some plugins/libraries only exist 64-bit versions like the Kraken method that requires oo2core_#_win64.dll (I think there are no 64-bit versions of these libraries).

3) I see in your examples that zlibwapi.dll is always with XTool.
Is this library necessary to be together with XTool in methods other than zlib?
I ask why I want to keep the libraries that it is not mandatory to always be with XTool in a subfolder and depending on the method selected for compression before compressing, a copy of these libraries will be made to XTool. After compressing, these copied libraries will be deleted. In the installer decompressors there will be all libraries used with XTool to compress all Data # .bin that will be extracted by the installer. So I asked the previous question (already answered), if I could have other libraries that were not used in the compression method with XTool when performing the extraction.

Thanks for the great job.