XTool 2020 (Main Project)

[elit]

Razor,

you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?

As for my idea,
It don't need to go through all 81 every time, only until match is found and then next stream can be tested with that level found in previous stream first. This means if streams are not mixed there will be no slowdown because first try is a match already. Also your own test implementation on first stream seem fast, why not use same technique, whatever is it doing?

I have never noticed ztool/xtool being slow at beginning, it seems to start immediately. Would there really be such big difference if it had to try 10 streams instead of 1, especially with optimization I mentioned above(starting with previously found level)? Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?

As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long).

You mentioned using stats, that sounds actually very similar to what I am talking about. In fact, if I understand correctly you may have already implemented what I am talking about here . But interestingly, you mentioned collecting stats from threads, that pretty much equals to cmp levels. So you in fact are collecting from multiple streams, not just first?

[Razor12911]

"you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?"
- it does for all streams, all 81 options but very quickly, takes milliseconds per stream to go through all 81 options.

"It don't need to go through all 81 every time"
- it doesn't go through all 81, sometimes even the first option it tries is correct and it stops, if it isn't correct then the 2nd one is most likely correct because of stats. XTool requests mode (value that appears most often), if that option fails, it requests mode in 2nd order (value that appears the 2nd most often) and so forth... which is why trials on all streams is necessary because it keeps the stats function filled with updated information about the current input.

"next stream can be tested with that level found in previous stream first."
- this method is the same as stat but it stores information limited to one stream which happens to be the previous stream, the approach of xtool is it stores a number of previous stream information, default is 256 streams.
Let's say for example
if the first stream was gave back level 9, the next stream is determined to be level 9, roughly the same as what you suggested, but what if the third stream is level 6? you have go through all the trials, ok fine, now you think then next stream is level 6 because of previous stream but what if that level 6 stream was just png image within the game data but instead the entire game was level 9? you try level 6 on the next stream it fails, then again you have to go through all 81 options again. XTool apporach thanks to stats, is level 9, level 9 and level 6 is what was stored, so for the next stream, it doesn't matter if the last one was level 6, it will try level 9 because it was detected twice compared to level 6, if it wasn't level 9, it will try that level 6 before it tries all 81 options because it was at least detected once, if that fails. it then tries 81 - 2 (6-9 from stats already failed) it does this for the past 256 streams and tries options in mode orders.

"I have never noticed ztool/xtool being slow at beginning, it seems to start immediately"
- It's because of stats plus other methods I've developed that make this possible.

"Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?"
You'll notice not just "few seconds of delay", yes few seconds for a set of streams but minutes of overall delay if you were to use this method for big streams, try this on dirt rally or doom 2016 and you'll see some incredibly low CPU utilization when using a lot of threads because other threads are obliged to wait for the first stream.

"As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long)."

with large streams, it is guaranteed to take long as mentioned before, which is why xtool faces too many problems with multi threading, I've instructed the threads to try 81 levels while using stats from all the streams they can find, as a result they end up fighting and crash each other it's hilarious but you get the idea, it's just best to try all levels on all streams but let stats decide in what order they should be presented in to speed up the process.

"You mentioned using stats, that sounds actually very similar to what I am talking about."
- Yes with the exception that not the previous stream option is considered but the previous 256 stream options are considered. If from those 256 options, there are 128 level 6s, 10 level 3s , 1 level 5 and 20 level 9s, it will try them in the order: 6 >> 9 >> 3 >> 5, then since all that failed, it begins trials, 1 - 9, it will skip, 3,5,6 and 9 because those failed, after that the new discovered level is added to stats, if it was 2, it will try the new order as 6 >> 9 >> 3 >> 5 >> 2, the reason all streams have to try all options is to get rid of non appearing options, example is when the data out of nowhere started to become level 4 out of nowhere, as stats are added, the number of level 9s will decrease because of these new levels 4s until they reach a certain threshold, then the order will now be 4 >> 6 >> 9 >> 3 >> 5 >> 2, as you can see level 4 over took level 6 but level 6 is still considered because before all of the level 4s there were a lot of level 6s so yes, I really did think deep about how to speed up xtool

"I understand correctly you may have already implemented what I am talking about here"
- Yep

"So you in fact are collecting from multiple streams, not just first?"
- Yes, I had to carefully do this else the threads crash themselves while fighting for the same stream or the same information, causes headaches but I find it funny when this happens, this is what causes most problems in pzlib/ztool and old xtool but if you properly implement it, you get to enjoy all the speed that comes with it.

[elit]

Finally, now(!) I am in picture. Thanks for taking your time, it does make a difference when I understand internals.

Believe or not until now I thought (z)xtool simply scan first stream and.. that's it - use same info for *all* rest of data lol . Reason I thought so was that game packs are likely using single specific level and that could in theory be enough(to pack the game, for which zxtools were designed).

Since that is not the case and your approach is clearly even much better than mine we can rest this case. Thanks again.

[Simorq]

1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work

decode has a bug.

decode:t1
Testing archive: data.arc
Tested 2 files, 4,023,182,817 => 2,147,912,601 bytes. Ratio 187.31%        
Testing time: cpu 1.09 sec/real 23.73 sec = 5%. Speed 90.52 mB/s
All OK

decode:t100p
Testing archive: data.arc
Tested 2 files, 4,023,182,817 => 2,147,912,601 bytes. Ratio 187.31%
Testing time: cpu 1.16 sec/real 22.47 sec = 5%. Speed 95.61 mB/s
All OK

Even with t1 it uses all cores!

[doofoo24]

Originally Posted by Simorq

1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work

also with x64 you can get "ERROR: write error (disk full?) in compression algorithm"...
i tested on doom with reflate t100p with c384mb i get ( ERROR: write error (disk full?))...
i changed to t4 it work...

for the decode it work i tested with t1 and t100p, for t1 use 11% of the cpu and for t100p it use all core...
***it's so important to emphasis do not use t100p for decode if you don't have a good cpu cooler, better to use t50p...

[Razor12911]

Guys, I’ll repeat. If you encounter an error, use xtool without fa so it produces an exception message so I know what the problem is

[Simorq]

#############################################
# reflate (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

xtool.exe precomp:reflate:c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1 = OK
Test 2 = OK
Test 3 = OK
Test 4 = OK

XTool x86 Command Line

xtool.exe precomp:reflate:c32mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

Test 1

P: OK
D: OK (FC: no differences encountered)

Test 2

P: EAccessViolation: Access violation at address 5378614D in module 'xtool.exe'. Read of address 5378614D
XTool is created by Razor12911

D: EReadError: Stream read error


Test 3

P: Exception EAccessViolation in module xtool.exe at 000BCEC7.
Access violation at address 004BCEC7 in module 'xtool.exe'. Read of address 00611D34.
D: EReadError: Stream read error

reflate works well with c16m in x86.

[Simorq]

#############################################
# zlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

zlib
xtool.exe precomp:zlib:c32mb and c16mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res


TEST 1
P: OK
D: CRC Error (0F2C892C: 09 D7)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: EAccessViolation: Access violation at address 00000000 in module 'xtool.exe'. Read of address 00000000

D: EReadError: Stream read error

zlib is very unstable, once successful, failed several times

[Simorq]

#############################################
# rzlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

XTool x86 Command Line

rzlib
xtool.exe precomp:rzlib:c16mb/c32mb/c64mb/c128mb,t100p - - < %1 > %1.out
xtool.exe decode:t100p - - < %1.out > %1.res
fc /b %1 %1.res

TEST 1
P: Error (EReadError: Stream read error)

TEST 2
P: Error (EReadError: Stream read error)

TEST 3
P: Error (EReadError: Stream read error)

TEST 4
P: Error (EReadError: Stream read error)

TEST 5
P: Error (EReadError: Stream read error)

rzlib always fails.

[Razor12911]

@Simorq

Looks like I'll have to go through debugging road once more

@everyone

Update available

1908_R3
- Compatiblity broken (old history db and inputs cannot work in this version)
- Added plugin support
- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)
- rzlib codec removed (use zlib)
- Added x86 memory limit

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins which can be found here

[doofoo24]

Originally Posted by Razor12911

- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)

"placeholders"

[ZAZA4EVER]

xtool_1908_R3 X64 ZLIB CODEC
eFootball Pes2020 Demo ... dt00_4K_x64.cpk

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 15,252,601 => 72,893,019 bytes. Ratio 477.91%
Compression time: cpu 0.05 sec/real 37.50 sec = 0%. Speed 0.41 mB/s
All OK

Zlib Codec is Stable .. i tested it Three Times ...
/////////////////////////////////////////////////////////////
xtool_1908_R3 X64 oodle Codec
Wolfenstein Youngblood ... chunkbase_9_pc.resources

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Test.bin
Compressed 1 file, 100,552,642 => 100,552,714 bytes. Ratio 100.00%
Compression time: cpu 0.33 sec/real 3.38 sec = 10%. Speed 29.79 mB/s
All OK

oodle leviathan Stream Dont Work

[FitGirl]

Originally Posted by ZAZA4EVER

oodle leviathan Stream Dont Work

They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea Also it's slow af in Oodle 2.6.* .

[Razor12911]

Originally Posted by ZAZA4EVER

oodle leviathan Stream Dont Work

Originally Posted by Razor12911

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins

Wolfenstein Youngblood ... chunkbase_9_pc.resources

- Could you upload a sample, I need to check something

Originally Posted by FitGirl

They have custom chunks in there, no size in headers, so the decompressor should guess the right size, which is not a good idea Also it's slow af in Oodle 2.6.* .

Last time I checked the headers did contain size, it was a while ago but I didn't add support because no game used this codec back then

[Simorq]

Hi Razor12911

Assassin's Creed: Origins
Test File: DataPC_extra.forge / oodle4

Creating archive: data.arc using oodle
Compressed 1 file, 1,049,427,968 => 1,049,428,104 bytes. Ratio 100.00%     
Compression time: cpu 1.47 sec/real 4.11 sec = 36%. Speed 255.62 mB/s
All OK

__________________________________________________ _____________________________________________
DataPC_SharedGroup_02.forge Sample File
Oodle_DLL_x86/x64

Oodle precompressor (Side project)

Creating archive: data.arc using oo2recm
Compressed 1 file, 105,512,960 => 167,404,655 bytes. Ratio 158.66%         
Compression time: cpu 0.17 sec/real 15.59 sec = 1%. Speed 6.77 mB/s
All OK

XTool 1908_R3

Creating archive: data2.arc using oodle
Compressed 1 file, 105,512,960 => 105,513,032 bytes. Ratio 100.00%         
Compression time: cpu 0.09 sec/real 0.53 sec = 18%. Speed 199.34 mB/s
All OK

______________________________________

The problem was solved thanks to KaktoR.