FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   XTool 2020 (Main Project) (https://fileforums.com/showthread.php?t=102832)

elit 27-12-2020 10:31

1 Attachment(s)
Exactly! But.. I doubt I am doing anything wrong, I mean:
Code:

-mcustom6/$gamelz=xtool:mcp2077+srp64+xlzma:lc8
*.archive = $gamelz group.
So *.archive = FA -mxtool:mcp2077+srp64+xlzma:lc8
or xtool precomp -t100p -mcp2077 to be precise
Code:

[External compressor:xtool]
header = 0
packcmd = _EC\xtool\xtool precomp -t100p {options} - - <stdin> <stdout>
 unpackcmd = _EC\xtool\xtool decode -t100p - - <stdin> <stdout>

Code:

srp64          = srep64:m3f:c256:mem8g:m64k:b16m:a2
procdefault    = srp64+xlzma:lc8

Code:

-mcustom6/$gamelz=xtool:mcp2077+procdefault
Anyone?(PS its latest 2012r2)
EDIT(restarted without t100p, but I don't think that's it: Attachment 28650
How you guys getting ~10mb/s is beyond me, are you tarring or doing individual files? As it may matter..)
EDIT2(speed down to 0.6mb/s, estimation ~22h+ and slowing down. Really, this ain't right but tool appear to be working correct not hanging! Can anyone try: latest xtool version with cp2077 plugin and on tarred archives through FA not individual files? I am sure there is something wrong in one of those variables.)
PS(I forgot to note that game is gog with 1.06 update, that means 9gb 1.05 was also applied, maybe it changed oodle structure? I will postpone compression until solution/cause is clear.)

elit 27-12-2020 11:52

So I just quickly tried oo2rect. That one seem to be working ok! I am going to use it on whole game and will see. I also tried xtool with -mkraken instead cp2077 plugin and also on individual archives. Speed issue remains. It does seem to have problem no matter what, while oo2rect seems fine.

EDIT:
oo2rect is same slow, after ~1h about 20gb and 0.xxmb/s speed. At this point I am pretty sure problem is not xtool nor oo2rect or plugin, but a game. If this was not a case with original version 1.03, try to update to 1.06, I think they changed data enough to become problem. For reference, files of concern are:
basegame_3_nightcity.archive
basegame_3_nightcity_gi.archive

elit 28-12-2020 03:09

Alright so I left it overnight. I can confirm it to work correctly but it took 11h:40min to compress, from that about ~10h due to xtool oodle. Archive tested ok and decompression was much quicker, compression is that slow though.

KaktoR 28-12-2020 03:34

No Idea what's wrong, but for me it's working just fine.

I use GOG version 1.06, along with an Ryzen 5 2600 and 16GB RAM and latest xtool version posted a few days ago.

I can test the whole game folder if you wish, but I can tell you the results will be just fine and nothing wrong.

Xtool settings I use:
Code:

[External compressor:xtool]
header    = 0
packcmd  = xtool.exe precomp -mcp2077 -c128mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

Code:

basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,714,137,974 bytes. Ratio 285.42%
Compression time: cpu 9.34 sec/real 704.44 sec = 1%. Speed 11.30 mB/s
All OK

Extracted 1 file, 22,714,137,974 => 7,958,224,896 bytes. Ratio 285.42%
Extraction time: cpu 8.11 sec/real 333.60 sec = 2%. Speed 23.86 mB/s
All OK


elit 28-12-2020 10:29

Hm, you have 6 core with hyperthreading. Maybe -t100p for you utilize all 12 virtual cores. You may try -t4 and also no --dbase to see if it affect speed that much. Also no need to test all archives only 2 files I mentioned above is enough. I wonder if --dbase or 12 virtual cores make for such difference. BTW thanks for the info above, it's interesting to know.

Razor12911 28-12-2020 23:30

i5 4590, 4 Cores (4 Threads)

Code:

[External compressor:xtool]
header    = 0
packcmd  = xtool.exe precomp -mcp2077 -c128mb -t100p - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

Code:

basegame_3_nightcity.archive

Compressed 1 file, 7,958,224,896 => 22,919,265,672 bytes. Ratio 287.99%
Compression time: cpu 9.91 sec/real 1538.09 sec = 1%. Speed 5.17 mB/s
All OK

Tested 1 file, 22,919,265,672 => 7,958,224,896 bytes. Ratio 287.99%
Testing time: cpu 6.45 sec/real 528.38 sec = 1%. Speed 15.06 mB/s
All OK

My concern is the difference in ratio, --dbase should not have an impact on ratio just the speed. Some bug I guess...

Masquerade 07-01-2021 05:47

Here's an interesting one.

ZZ1.dat - 175MB from Steel Division 2.


XTool 2020 (zlib)
Code:

Compressed 1 file, 183,623,680 => 183,661,889 bytes. Ratio 100.02%
Compression time: cpu 0.16 sec/real 26.08 sec = 1%. Speed 7.04 mB/s
All OK

Xtool v12 (zlib)
Code:

Compressed 1 file, 183,623,680 => 226,689,420 bytes. Ratio 123.45%
Compression time: cpu 0.27 sec/real 26.18 sec = 1%. Speed 7.01 mB/s
All OK

XTool v12 hangs inifintely when during decompression - the file just keeps getting bigger and bigger.

GFS Detects zlib streams, whereas Drop Scan does not:

https://i.imgur.com/YHG8hpl.png

https://i.imgur.com/6n0GrEA.png

https://anonfiles.com/FcY1Q060p9/ZZ_1_dat

KaktoR 07-01-2021 05:50

Make headerless+force detection

PS: The anonfiles is shit. Always slow connection here :D (185kb/s)

KaktoR 07-01-2021 06:02

@Masquerade
Use reflate codec

Code:

Compressed 1 file, 183,623,680 => 227,188,297 bytes. Ratio 123.72%
Compression time: cpu 0.09 sec/real 5.12 sec = 2%. Speed 35.87 mB/s
All OK

Extracted 1 file, 227,188,297 => 183,623,680 bytes. Ratio 123.72%
Extraction time: cpu 0.13 sec/real 1.07 sec = 12%. Speed 171.41 mB/s
All OK


Razor12911 07-01-2021 18:59

When you use zlib method in Xtool v12 or older variants of xtool, they automatically used reflate when you placed the libraries each time zlib fails, the new xtool does not do any of this. The user is the one who needs to do this by themselves because what I have noticed is people putting reflate libraries near xtool and they never know when xtool uses them or when it doesn't. This way the user knows what the issues is when decompression fails each time when they try to figure out what the actual problem is.

I also did point out that the best method for precompressing anything that is zlib/deflate compressed, use -mzlib along with either reflate/preflate so you get the best results.

Masquerade 08-01-2021 05:14

Razor12911,
I did use zlib+preflate, I simply forgot about reflate.

After using relate, everything works fine.

Razor12911 08-01-2021 20:00

Update available

Changes

- updated library support
- updated command line parser
- included x86 build
- fixed depthing issues

Notes

I have added 32-bit build as requested but you'll have to use x86 libraries or find a matching pair of x86/x64 libraries. You can get these from github of a project under the releases section.
The versioning is brought back to the format of 1.0.0 instead of the build number as requested.

Masquerade 09-01-2021 02:31

@Razor12911

Thanks for this new method... incredible!

Code:

Compressing 1 file, 17,530,888,054 bytes
  Compressing TSCGame-WindowsNoEditor.pak
Compressed 1 file, 17,530,888,054 => 28,460,607,223 bytes. Ratio 162.35%
Compression time: cpu 18.42 sec/real 624.55 sec = 3%. Speed 28.07 mB/s
All OK

On the sample I sent to you yesterday.

Cesar82 09-01-2021 08:03

@Razor12911, thanks for the 32-bit version:eek:, it will be very useful to include in CIU.
1) Also like to inform you that it is being detected as a false positive by the KIS anti-virus (I don't know if there is anything that can be done).
P.S: The 64-bit version has not been captured.
https://i.imgur.com/rdaSySe.png
2) You could share libraries compatible with the 32-bit version to be fully compatible and avoid errors due to incompatibilities.
3) I wonder if I have 3 games (collection) and each game uses XTool with different methods to compress.
Assuming that only the libraries needed for compression (nothing more than necessary) for each Data.bin file are together with XTool.exe at the time of compressing.
To perform the decompression, can I have other libraries that were not used to compress the Data.bin file together with XTool (Like: Have 1 library next to XTool when compressing and 5 when decompressing)?
Thanks!

Razor12911 09-01-2021 08:34

1 Attachment(s)
1) some interesting stuff,
x86: https://www.virustotal.com/gui/file/...296d/detection
x64: https://www.virustotal.com/gui/file/...7dfe/detection

the code is actually the same so I don't know... :D

2) Yes the issue is also the plugins which also have to shipped with 32-bit version, who still uses 32-bit system anyways?
Attachment 28717

3) Yes, shouldn't be a problem.

I'll ship x86 libraries but any incompatibility issues should fall on the end user.


All times are GMT -7. The time now is 09:58.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com