FileForums - XTool - Successor of ZTool

Page 18 of 39

« First

Last »

Show 40 post(s) from this thread on one page

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - XTool - Successor of ZTool (https://fileforums.com/showthread.php?t=101729)

ZakirAhmad

30-05-2018 08:52

xtool v 0.7 Tried it on a file from AC orgins

it didnt expanded.
sorry to bother oodle was missing from arc.ini setting.
this sorted the issue.

afr expaned it to 163.15% while xtool to 162.91% xtool
restored it in only 14sec while afr took 18 seconds in my system

Now my only problem is with Dunia engine. i wrote dat file address in xtool.ini still no change.

BTW is farcry primal and James Camerons avatar supported.

KaktoR

30-05-2018 09:15

Yep, didn't expand with :lzo

Maybe wrong codec in arc.ini? lzo2.dll is in place

Edit: You have to use

Code:

:oodle

for Origins:rolleyes::D

Code:

DataPC.forge (ACO)



xtool v0.7 oodle: 273.088.512 Bytes => 574.853.120 Bytes

AFR:              273.088.512 Bytes => 574.849.024 Bytes

From this example xtool and AFR have pretty much the same compression speed (AFR is a little bit faster, but just a few secs)

elit	30-05-2018 13:52

Quote:

Originally Posted by Razor12911 (Post 470916)

Update available
In terms of the Dunia engine, I advise that you use the latest liblz4.dll to avoid some streams being skipped due to their major difference, this could be because even I myself am not sure if I used the correct parameters for recompression, if you have an idea of what is used, please comment. (I used lz4hc, level 9)

I dont know if you want to go this route, but the way I did my tests when developing lz4 packer for "Raiders of the Broken Planet" game was that I used both original and self-modified quickbms script for unpacking game archive first. I unpacked thus twice - for each.

In modified script only difference was that I disabled decompression by taking out "comtype" command and used "log" instead of "clog" to dump files raw & compressed.

So I had both compressed and decompressed individual files from archive. Then it was only matter of finding right/best cmp settings through hex editor and binary comparison(first on few files and then all) to get best setting. And even wrong(but close) settings were still compatible with 80%+ of all archive.

ShivShubh

30-05-2018 20:42

Quote:

Originally Posted by elit (Post 471297)

So by saying all that what were you trying to say by this "I dont know if you want to go this route" ? That Razor should find the correct parameters ? I don't think he has that much of an interest in far cry 5 or wants to waste time doing all that.

Razor12911

30-05-2018 21:28

Quote:

Originally Posted by Sergey3695 (Post 471284)

https://yadi.sk/d/WNKW73Ht3UpyVN
data.cpk (393 mb)
NARUTO SHIPPUDEN Ultimate Ninja STORM 2

[External compressor:xprecomp]
header = 0
packcmd = xtool.exe e:precomp:c32mb,t1:crilayla - - <stdin> <stdout>
unpackcmd = xtool.exe d:precomp:c32mb,t1:crilayla - - <stdin> <stdout>

Extracting archive: data.arc
Extracting data.cpk (413123988 bytes)
ERROR: file _TEST\data.cpk failed CRC check
:confused:

i'll check :)

Edison007

31-05-2018 03:20

compr:

Code:

afr:v0+srep:m5f:l512:d512m:hash-+lolz:d64m - 37,6 MB (39 463 094 bytes)

xtool+srep:m5f:l512:d512m:hash-+lolz:d64m  - 37,8 MB (39 671 049 bytes)

dec time/speed:

Code:

cls-afr, v019 (x86)

t1 - 199.28 sec; ~  5.37 mB/s

t7 -  44.07 sec; ~ 24.29 mB/s

t8 -  42.89 sec; ~ 24.96 mB/s



xtool_x86, v07

t1 - 237.07 sec; ~  4.52 mB/s

t7 -  53.84 sec; ~ 19.88 mB/s

t8 -  51.19 sec; ~ 20.91 mB/s



xtool_x64, v07

t1 - 210.85 sec; ~  5.08 mB/s

t7 -  49.35 sec; ~ 21.69 mB/s

t8 -  46.91 sec; ~ 22.82 mB/s

Code:

1'070'530'560 ->   afr: 1'767'925'585 bytes

              -> xtool: 1'768,021'857 bytes



i7-4700MQ, 8gb ram, ram-disk, win7x64.

shazzla

31-05-2018 04:14

Similar happens on Agony's .PAK files(zlib,v0.7, v0.6 works good)
First file processed well,but the second one looks like simply copied.
Cant upload files because they are more than 10 gigs. :/

elit	31-05-2018 08:50

Quote:

Originally Posted by doofoo24 (Post 471309)

i think it may be better for Razor12911 if he develop the tool separately like pzlib/plz4/pzstd/plzo for games...
less of a headache ;)

How exactly is that less headache? This way at least he can re-use a lot of same code/routines for each compressor instead of duplicating and maintaining separate projects with a lot of same code.

Razor12911

31-05-2018 09:31

Update available

Changes

- fixed crilayla bug on naruto games
- fixed issue detecting lzo2a, lzo1c streams

@doofoo24
I have to agree with elit, It's more headache separating the codecs. An example would be ztool itself, normally when I make changes to pzlib, if it's not related to precompression itself, I had to make the same changes in plz4, plzo... sometimes I forget doing this and end up with different sources of each, in one, a bug was fixed then another bug appears in another, now there are bugs all over the codecs, different bugs, you end up trying to fix one and another pops up and so forth. The idea of putting everything together means one code, if one codec is affected, then all will be affected then I'll know the source of the bug.

Quote:

Originally Posted by Edison007 (Post 471306)

compr:

Code:

afr:v0+srep:m5f:l512:d512m:hash-+lolz:d64m - 37,6 MB (39 463 094 bytes)

xtool+srep:m5f:l512:d512m:hash-+lolz:d64m  - 37,8 MB (39 671 049 bytes)

dec time/speed:

Code:

cls-afr, v019 (x86)

t1 - 199.28 sec; ~  5.37 mB/s

t7 -  44.07 sec; ~ 24.29 mB/s

t8 -  42.89 sec; ~ 24.96 mB/s



xtool_x86, v07

t1 - 237.07 sec; ~  4.52 mB/s

t7 -  53.84 sec; ~ 19.88 mB/s

t8 -  51.19 sec; ~ 20.91 mB/s



xtool_x64, v07

t1 - 210.85 sec; ~  5.08 mB/s

t7 -  49.35 sec; ~ 21.69 mB/s

t8 -  46.91 sec; ~ 22.82 mB/s

Code:

1'070'530'560 ->   afr: 1'767'925'585 bytes

              -> xtool: 1'768,021'857 bytes



i7-4700MQ, 8gb ram, ram-disk, win7x64.

I wonder if there is still room for improvement.

Razor12911

31-05-2018 09:52

Quote:

Originally Posted by doofoo24 (Post 471315)

@Razor12911 xtool 0.7 seems to skip file's test it on ME A with zstd and before with lzo on ac 1...
xtool 0.6 work fine...
any idea ?
i will test with 0.8...

I didn't touch zstd codec in between the updates

ztool_08

Code:

Process ID       : 12240

Thread ID        : 19080

Process Exit Code: 0

Thread Exit Code : 0



User Time        :          24.500s

Kernel Time      :           0.734s

Process Time     :          25.234s

Clock Time       :           7.764s



Working Set      :          193016 KB

Paged Pool       :             114 KB

Nonpaged Pool    :              13 KB

Pagefile         :          314972 KB

Page Fault Count : 311084



IO Read          :           43332 KB (in             683 reads )

IO Write         :          101353 KB (in            1584 writes)

IO Other         :               3 KB (in             178 others)

ztool_06

Code:

Process ID       : 12748

Thread ID        : 14488

Process Exit Code: 0

Thread Exit Code : 0



User Time        :          24.890s

Kernel Time      :           1.125s

Process Time     :          26.015s

Clock Time       :           8.160s



Working Set      :          177480 KB

Paged Pool       :             115 KB

Nonpaged Pool    :              12 KB

Pagefile         :          229352 KB

Page Fault Count : 316978



IO Read          :           43262 KB (in             677 reads )

IO Write         :          101353 KB (in            1584 writes)

IO Other         :               5 KB (in             270 others)

Razor12911

31-05-2018 10:02

1 Attachment(s)

Quote:

Originally Posted by Edison007 (Post 471306)

compr:

Code:

afr:v0+srep:m5f:l512:d512m:hash-+lolz:d64m - 37,6 MB (39 463 094 bytes)

xtool+srep:m5f:l512:d512m:hash-+lolz:d64m  - 37,8 MB (39 671 049 bytes)

dec time/speed:

Code:

cls-afr, v019 (x86)

t1 - 199.28 sec; ~  5.37 mB/s

t7 -  44.07 sec; ~ 24.29 mB/s

t8 -  42.89 sec; ~ 24.96 mB/s



xtool_x86, v07

t1 - 237.07 sec; ~  4.52 mB/s

t7 -  53.84 sec; ~ 19.88 mB/s

t8 -  51.19 sec; ~ 20.91 mB/s



xtool_x64, v07

t1 - 210.85 sec; ~  5.08 mB/s

t7 -  49.35 sec; ~ 21.69 mB/s

t8 -  46.91 sec; ~ 22.82 mB/s

Code:

1'070'530'560 ->   afr: 1'767'925'585 bytes

              -> xtool: 1'768,021'857 bytes



i7-4700MQ, 8gb ram, ram-disk, win7x64.

Ran a test using the samples I got from forum. Very small input but got these results:

Input: 278 MB (291,605,504 bytes)
DataPC_Map_Menu.forge (ACI)
DataPC_LoadingRoom2.forge (ACI)
DataPC.forge (ACII)
DataPC.forge (ACIV)

Quote:

command line:
xtool_x86: d:precomp:t# %1.out1 %1.res
afr19_x86: d -v0 -t# %1.out2 %1.res

Thread(1) = 1 result:
xtool: 17.898s, 18.050s, 17.901s, 18.056s, 17.942s
afr19: 17.853s, 17.875s, 18.173s, 17.865s, 17.879s

Thread(s) = 2 result:
xtool: 9.120s, 9.156s, 9.035s, 9.067s, 9.008s
afr19: 9.394s, 9.356s, 9.318s, 9.248s, 9.291s

Thread(s) = 3 result:
xtool: 6.325s, 6.412s, 6.249s, 6.412s, 6.264s
afr19: 6.603s, 6.511s, 6.696s, 6.517s, 6.691s

Not sure in terms of results here, I ran test on ramdisk using my old core 2 extreme, I didn't include t4 since those are mostly irrelevant.

As for final output after compression, I'd be taking a gamble when I say it's headers since they include stuff like what codec was used since xtool is not just made for one game engine but I'll check in terms of the number of streams found by xtool vs afr

Razor12911

31-05-2018 10:08

Quote:

Originally Posted by doofoo24 (Post 471319)

tried xtool 0.8 and 0.7 on multiple files at once like shazzla First file processed well,but the second one looks like simply copied.
lz4 / lzo /zlib

upload files and command line used

pakrat2k2

31-05-2018 10:09

ALWAYS choose disable smilies, when posting/ replying ( checkbox below message box )

pain to have to come in & clean up posts & quotes.

Edison007

31-05-2018 10:40

Quote:

Not sure in terms of results here

It's normal, because i using differents decode methods in cls and exe. In cls i use some tricks, which allow me to get a higher speed. exe-decoder, yes, slower.

Quote:

I'll check in terms of the number of streams found by xtool vs afr

I think, that numbers of streams found is the same. But maybe in AFR the output container is more successfully made ;)

Quote:

I wonder if there is still room for improvement.

Yep, but I'm not interested in this engine anymore.

KaktoR

31-05-2018 10:56

Quote:

Originally Posted by Edison007 (Post 471325)

Yep, but I'm not interested in this engine anymore.

So, what's your next clue? :D