[Dev]XTool - Page 4

Gupta · #46 05-04-2018, 15:26

x64w.rpf(checked twice)

Code:

Process ID       : 4960
Thread ID        : 2752
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         299.953s
Kernel Time      :           1.515s
Process Time     :         301.468s
Clock Time       :         128.594s

Working Set      :          344464 KB
Paged Pool       :              93 KB
Nonpaged Pool    :              13 KB
Pagefile         :          373868 KB
Page Fault Count : 358433

IO Read          :          915372 KB (in           14304 reads )
IO Write         :         2078066 KB (in           32470 writes)
IO Other         :               1 KB (in             138 others)


Process ID       : 7556
Thread ID        : 7144
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         315.703s
Kernel Time      :           2.078s
Process Time     :         317.781s
Clock Time       :         110.843s

Working Set      :          443228 KB
Paged Pool       :              92 KB
Nonpaged Pool    :              13 KB
Pagefile         :          497848 KB
Page Fault Count : 496542

IO Read          :          915372 KB (in           14305 reads )
IO Write         :         2073414 KB (in           32399 writes)
IO Other         :               1 KB (in             120 others)

Code:

x64w.rpf.out1.lzma: 2,127,939,711 -> 595,208,759: 27.97%  
x64w.rpf.out2.lzma: 2,123,176,204 -> 598,727,541: 28.20%

School.tar(~30 pdfs, 200kB to 150mB)

Code:

Process ID       : 9736
Thread ID        : 896
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         300.541s
Kernel Time      :         103.625s
Process Time     :         404.166s
Clock Time       :         431.191s

Working Set      :         1053160 KB
Paged Pool       :              93 KB
Nonpaged Pool    :              13 KB
Pagefile         :         1171552 KB
Page Fault Count : 86155222

IO Read          :          581265 KB (in            9084 reads )
IO Write         :         2164138 KB (in           33815 writes)
IO Other         :               1 KB (in             210 others)


Process ID       : 2492
Thread ID        : 5196
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         365.791s
Kernel Time      :         125.078s
Process Time     :         490.869s
Clock Time       :         486.174s

Working Set      :         1157188 KB
Paged Pool       :              92 KB
Nonpaged Pool    :              42 KB
Pagefile         :         1281496 KB
Page Fault Count : 98349470

IO Read          :          581265 KB (in            9085 reads )
IO Write         :         2173087 KB (in           33956 writes)
IO Other         :               1 KB (in             148 others)

@KaktoR

Quote:

User Time : 48.187s
Kernel Time : 0.859s
Process Time : 49.046s
Clock Time : 31.565s

why ztool so fast here or it crashed in testing?

KaktoR · #47 05-04-2018, 15:54

@Gupta: It crashed

KaktoR · #48 05-04-2018, 17:09

Quote:

Originally Posted by Razor12911

Thanks for the tests guys. Here's another one, I added reflate lib support on this test, the inputs I'm interested in is data where reflate is required otherwise it will be just the same results as test number 4.

Png images, pdfs, linux isos and zip files contain streams that can be processed by reflate, games however that's DOOM and Dishonored 2 afaik.

So just like test 4, post cmd output.

I just like to point out, I'm not interested in the size that much but memory used and the time it took to perform the test.

These are the results of the tests I have performed:
Attachment 21616

I can't tell you how much RAM it used (where i can look? TM is quiet impractical).

*.png files (~8k) all i could find on HDD (except 20k+ from sysdrive). I packed them in a stored m0 zip file (however streams are the same) just because your batch can't process a folder, only a single file.
366 MB (384.172.032 Bytes)

Code:

Process ID       : 9160
Thread ID        : 4300
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         384.144s
Kernel Time      :          33.265s
Process Time     :         417.409s
Clock Time       :         856.020s

Working Set      :         1435080 KB
Paged Pool       :             100 KB
Nonpaged Pool    :              12 KB
Pagefile         :         1590936 KB
Page Fault Count : 24544849

IO Read          :          364298 KB (in            5694 reads )
IO Write         :         3283916 KB (in           51312 writes)
IO Other         :               5 KB (in             461 others)


Process ID       : 7896
Thread ID        : 1392
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         338.692s
Kernel Time      :          84.125s
Process Time     :         422.817s
Clock Time       :        1614.952s

Working Set      :         1589080 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              87 KB
Pagefile         :         1844716 KB
Page Fault Count : 64115442

IO Read          :          364298 KB (in            5695 reads )
IO Write         :         3296645 KB (in           51512 writes)
IO Other         :               5 KB (in             418 others)

*.pdf files (16 files). Same as png, packed with m0 in zip file
256 MB (269.295.616 Bytes)

Code:

Process ID       : 3904
Thread ID        : 5204
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         300.359s
Kernel Time      :          23.937s
Process Time     :         324.296s
Clock Time       :          97.433s

Working Set      :          473056 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              13 KB
Pagefile         :          526072 KB
Page Fault Count : 17154634

IO Read          :          262959 KB (in            4111 reads )
IO Write         :          564426 KB (in            8820 writes)
IO Other         :               2 KB (in             300 others)


Process ID       : 6356
Thread ID        : 6364
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         399.656s
Kernel Time      :          28.109s
Process Time     :         427.765s
Clock Time       :         128.859s

Working Set      :          574620 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              26 KB
Pagefile         :          644664 KB
Page Fault Count : 20138944

IO Read          :          262959 KB (in            4112 reads )
IO Write         :          566978 KB (in            8861 writes)
IO Other         :               2 KB (in             148 others)

------------------------------------------------------------------------------

Single PNG file
15 MB (15.794.176 Bytes)

Code:

Process ID       : 5764
Thread ID        : 2916
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          23.031s
Kernel Time      :           0.125s
Process Time     :          23.156s
Clock Time       :          23.248s

Working Set      :           74904 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              10 KB
Pagefile         :          161384 KB
Page Fault Count : 38067

IO Read          :           15422 KB (in             242 reads )
IO Write         :           34757 KB (in             544 writes)
IO Other         :               2 KB (in             170 others)


Process ID       : 5784
Thread ID        : 10224
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         115.343s
Kernel Time      :           0.687s
Process Time     :         116.030s
Clock Time       :         116.374s

Working Set      :          180072 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              12 KB
Pagefile         :          200228 KB
Page Fault Count : 600207

IO Read          :           15422 KB (in             243 reads )
IO Write         :           36515 KB (in             572 writes)
IO Other         :               2 KB (in             136 others)

Single PDF file (The_Art_Of_SGWarlords.pdf)
45.8 MB (48.070.656 Bytes)

Code:

Process ID       : 5056
Thread ID        : 10140
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          83.156s
Kernel Time      :           1.140s
Process Time     :          84.296s
Clock Time       :          23.200s

Working Set      :          332048 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              13 KB
Pagefile         :          424816 KB
Page Fault Count : 1070466

IO Read          :           46941 KB (in             735 reads )
IO Write         :          263631 KB (in            4120 writes)
IO Other         :               2 KB (in             170 others)


Process ID       : 3472
Thread ID        : 484
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         250.109s
Kernel Time      :           4.390s
Process Time     :         254.499s
Clock Time       :          73.490s

Working Set      :          469956 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              28 KB
Pagefile         :          536544 KB
Page Fault Count : 5065503

IO Read          :           46941 KB (in             736 reads )
IO Write         :          265579 KB (in            4151 writes)
IO Other         :               2 KB (in             138 others)

Simorq · #49 05-04-2018, 19:17

DMCL-0.nbz (165 MB)

Code:

Process ID       : 6860
Thread ID        : 5652
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          72.450s
Kernel Time      :           9.078s
Process Time     :          81.528s
Clock Time       :         577.212s

Working Set      :          369332 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              12 KB
Pagefile         :          442580 KB
Page Fault Count : 3711470

IO Read          :          168982 KB (in            2642 reads )
IO Write         :          548729 KB (in            8574 writes)
IO Other         :               5 KB (in             293 others)


Process ID       : 6420
Thread ID        : 2940
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         188.023s
Kernel Time      :          30.140s
Process Time     :         218.163s
Clock Time       :        1565.767s

Working Set      :          440868 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              32 KB
Pagefile         :          501228 KB
Page Fault Count : 11514042

IO Read          :          168982 KB (in            2643 reads )
IO Write         :          573538 KB (in            8963 writes)
IO Other         :               5 KB (in             304 others)

Dmc2-0.nbz (409 MB)

Andu21 · #50 05-04-2018, 21:32

TombRaider_Beginning.pdf - 904 MB (948,140,632 bytes)

Code:

Process ID       : 3144
Thread ID        : 2536
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         129.174s
Kernel Time      :           3.619s
Process Time     :         132.793s
Clock Time       :         409.011s

Working Set      :          914324 KB
Paged Pool       :              85 KB
Nonpaged Pool    :              11 KB
Pagefile         :         1008020 KB
Page Fault Count : 1268744

IO Read          :          925918 KB (in           14469 reads )
IO Write         :         4782205 KB (in           74722 writes)
IO Other         :               1 KB (in             172 others)


Process ID       : 3116
Thread ID        : 2036
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         357.273s
Kernel Time      :          99.856s
Process Time     :         457.129s
Clock Time       :        6069.699s

Working Set      :         1422572 KB
Paged Pool       :              85 KB
Nonpaged Pool    :              14 KB
Pagefile         :         1632072 KB
Page Fault Count : 92402768

IO Read          :          925918 KB (in           14469 reads )
IO Write         :         4899729 KB (in           76559 writes)
IO Other         :               1 KB (in             121 others)

Code:

TombRaider_Beginning.pdf.out1 - 4.56 GB (4,896,978,011 bytes)
TombRaider_Beginning.pdf.out2 - 4.67 GB (5,017,323,321 bytes)

Razor12911 · #51 05-04-2018, 23:13

@PrinceGupta
Turns out I left dynamic entropy calculator in the scanner, as a result it slowed down the scanner but re-ran the test without the entropy function and this came:

x64w.rpf

Code:

Process ID       : 9004
Thread ID        : 8760
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         277.531s
Kernel Time      :           2.890s
Process Time     :         280.421s
Clock Time       :          85.264s

Working Set      :          398088 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              12 KB
Pagefile         :          446216 KB
Page Fault Count : 377597

IO Read          :          915376 KB (in           14305 reads )
IO Write         :         2077976 KB (in           32469 writes)
IO Other         :               2 KB (in             200 others)


Process ID       : 20692
Thread ID        : 17040
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         306.671s
Kernel Time      :           3.093s
Process Time     :         309.764s
Clock Time       :          93.089s

Working Set      :          453512 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              13 KB
Pagefile         :          509212 KB
Page Fault Count : 422732

IO Read          :          915376 KB (in           14306 reads )
IO Write         :         2073414 KB (in           32399 writes)
IO Other         :               2 KB (in             155 others)

@Simorq

Downloaded the file and seems to have worked fine here:

dmc2-0.nbz

Code:

Process ID       : 18168
Thread ID        : 12628
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          58.462s
Kernel Time      :          30.437s
Process Time     :          88.899s
Clock Time       :         388.581s

Working Set      :          481856 KB
Paged Pool       :              98 KB
Nonpaged Pool    :              13 KB
Pagefile         :          533624 KB
Page Fault Count : 12609675

IO Read          :          419569 KB (in            6558 reads )
IO Write         :         1482029 KB (in           23157 writes)
IO Other         :               4 KB (in             393 others)


Process ID       : 11436
Thread ID        : 3180
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          35.757s
Kernel Time      :          70.593s
Process Time     :         106.350s
Clock Time       :        1520.643s

Working Set      :          544004 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              32 KB
Pagefile         :          611236 KB
Page Fault Count : 28470168

IO Read          :          419579 KB (in            6559 reads )
IO Write         :         1511624 KB (in           23621 writes)
IO Other         :               3 KB (in             304 others)

Press any key to continue . . .

Webster17 · #52 05-04-2018, 23:33

Quote:

Originally Posted by Razor12911

Thanks for the tests guys. Here's another one, I added reflate lib support on this test, the inputs I'm interested in is data where reflate is required otherwise it will be just the same results as test number 4.

Png images, pdfs, linux isos and zip files contain streams that can be processed by reflate, games however that's DOOM and Dishonored 2 afaik.

So just like test 4, post cmd output.

I just like to point out, I'm not interested in the size that much but memory used and the time it took to perform the test.

These are the results of the tests I have performed:
Attachment 21616

Test 3 took seconds to recompress Darksiders 2 file anim_streams.upak but test 5 took more like precomp and the output file is smaller than test 3 output, BTW if you are "not interested in the size that much" why build it in the first place?

Webster17 · #53 05-04-2018, 23:36

oh and sorry i couldn't participate in the tests, i was sick just recovered today

Razor12911 · #54 06-04-2018, 15:06

Quote:

Originally Posted by Simorq

DMCL-0.nbz (165 MB)

Code:

Process ID       : 6860
Thread ID        : 5652
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          72.450s
Kernel Time      :           9.078s
Process Time     :          81.528s
Clock Time       :         577.212s

Working Set      :          369332 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              12 KB
Pagefile         :          442580 KB
Page Fault Count : 3711470

IO Read          :          168982 KB (in            2642 reads )
IO Write         :          548729 KB (in            8574 writes)
IO Other         :               5 KB (in             293 others)


Process ID       : 6420
Thread ID        : 2940
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         188.023s
Kernel Time      :          30.140s
Process Time     :         218.163s
Clock Time       :        1565.767s

Working Set      :          440868 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              32 KB
Pagefile         :          501228 KB
Page Fault Count : 11514042

IO Read          :          168982 KB (in            2643 reads )
IO Write         :          573538 KB (in            8963 writes)
IO Other         :               5 KB (in             304 others)

Dmc2-0.nbz (409 MB)

from what game are these files coming from?

Simorq · #55 06-04-2018, 15:12

Razor12911

Devil May Cry HD Collection

Razor12911 · #56 06-04-2018, 15:17

Quote:

Originally Posted by Simorq

Razor12911

Devil May Cry HD Collection

Well it seems like the scanner is improved.

dmc2-0.nbz:
xtool: 409 MB > 1.48 GB > 162 MB
ztool: 409 MB > 1.44 GB > 181 MB

DMCL-0.nbz:
xtool: 165 MB > 617 MB > 130 MB
ztool: 165 MB > 560 MB > 148 MB

I have to run more tests though to confirm this.

Simorq · #57 06-04-2018, 15:35

animations-part1.pak (334 MB) (Kingdom Come)

Code:

Process ID       : 4524
Thread ID        : 7792
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          75.093s
Kernel Time      :           0.796s
Process Time     :          75.889s
Clock Time       :          51.127s

Working Set      :          501300 KB
Paged Pool       :              97 KB
Nonpaged Pool    :              13 KB
Pagefile         :          554228 KB
Page Fault Count : 336186

IO Read          :          342891 KB (in            5359 reads )
IO Write         :          561879 KB (in            8780 writes)
IO Other         :               4 KB (in             293 others)


Process ID       : 7052
Thread ID        : 2176
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         301.703s
Kernel Time      :           6.234s
Process Time     :         307.937s
Clock Time       :         284.318s

Working Set      :          617036 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              14 KB
Pagefile         :          674440 KB
Page Fault Count : 3790034

IO Read          :          342891 KB (in            5360 reads )
IO Write         :          561876 KB (in            8781 writes)
IO Other         :               4 KB (in             268 others)

Code:

animations-part1.pak
XTool: 334 MB > 548 MB > 301 MB
ZTool: 334 MB > 548 MB > 301 MB

Simorq · #58 06-04-2018, 16:02

Asset2.bra (475 MB) (Tokyo Xanadu eX)

Code:

Process ID       : 5792
Thread ID        : 4656
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         251.675s
Kernel Time      :           9.171s
Process Time     :         260.846s
Clock Time       :         202.494s

Working Set      :          247496 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              12 KB
Pagefile         :          268176 KB
Page Fault Count : 3418788

IO Read          :          487304 KB (in            7616 reads )
IO Write         :          839258 KB (in           13114 writes)
IO Other         :               1 KB (in             165 others)


Process ID       : 5656
Thread ID        : 2980
Process Exit Code: 0
Thread Exit Code : 0

User Time        :         154.153s
Kernel Time      :          31.609s
Process Time     :         185.762s
Clock Time       :         571.207s

Working Set      :          335744 KB
Paged Pool       :              95 KB
Nonpaged Pool    :              20 KB
Pagefile         :          383708 KB
Page Fault Count : 12619767

IO Read          :          487304 KB (in            7617 reads )
IO Write         :          838611 KB (in           13105 writes)
IO Other         :               1 KB (in             148 others)

Code:

Asset2.bra
XTool: 475 MB > 819 MB > 315 MB (330,378,763 Bytes)
ZTool: 475 MB > 818 MB > 315 MB (330,628,132 Bytes)

Razor12911 · #59 06-04-2018, 16:03

Nice, about 5 times faster

while using a bit less memory

Simorq · #60 06-04-2018, 16:10

EAC.tat (229 MB) (226 EAC file of Final Fantasy XV)

Code:

Process ID       : 7912
Thread ID        : 5432
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          19.859s
Kernel Time      :           0.562s
Process Time     :          20.421s
Clock Time       :           9.266s

Working Set      :          164912 KB
Paged Pool       :              96 KB
Nonpaged Pool    :              11 KB
Pagefile         :          189484 KB
Page Fault Count : 241612

IO Read          :          235237 KB (in            3677 reads )
IO Write         :          257963 KB (in            4031 writes)
IO Other         :               1 KB (in             165 others)


Process ID       : 8056
Thread ID        : 5940
Process Exit Code: 0
Thread Exit Code : 0

User Time        :          30.250s
Kernel Time      :           0.671s
Process Time     :          30.921s
Clock Time       :           9.985s

Working Set      :          221512 KB
Paged Pool       :              95 KB
Nonpaged Pool    :              13 KB
Pagefile         :          253772 KB
Page Fault Count : 318112

IO Read          :          235237 KB (in            3678 reads )
IO Write         :          257915 KB (in            4031 writes)
IO Other         :               1 KB (in             131 others)

Code:

EAC.tat
XTool: 229 MB > 251 MB > 218 MB (229,372,585 Bytes)
ZTool: 229 MB > 251 MB > 218 MB (229,571,558 Bytes)

The Following User Says Thank You to Gupta For This Useful Post:
Razor12911 (05-04-2018)

The Following User Says Thank You to KaktoR For This Useful Post:
Gupta (05-04-2018)

The Following User Says Thank You to Simorq For This Useful Post:
Razor12911 (05-04-2018)

The Following User Says Thank You to Andu21 For This Useful Post:
Razor12911 (05-04-2018)

The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
ShivShubh (24-11-2019), Simorq (05-04-2018)

The Following User Says Thank You to Razor12911 For This Useful Post:
ShivShubh (24-11-2019)

The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
ShivShubh (24-11-2019), Simorq (06-04-2018)

The Following User Says Thank You to Simorq For This Useful Post:
Razor12911 (06-04-2018)