Can you run xtool outside freearc, like via console.

I'm interested in the exception/error message

@Razor12911
good
http://s8.picofile.com/file/8368124442/rrrr.jpg

-------------------------
I think the main problem is - -.
- - <stdin> <stdout>:confused:

try this then

@Razor12911
no work
stuck cpu
http://s9.picofile.com/file/8368124692/rrr.jpg

Nice ghost bugs, I'll check what I missed in the source and thanks for the tests.

I have a problem using t100p.

add pls lvl setting for reflate 1-9. thx.

Have patience

using these parameters, the expansion reaches 2.4% and freezes.

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp:rzlib,zlib,reflate:c16mb,t2 - - <stdin> <stdout>


I am using the x64 version.

And why do you use all three methods at the same time?

если я оставлю только один метод, ошибка та же.

----

if I leave only one method, the error is the same.

But what is the difference between rzlib and zlib?

Don't be that lazy.
https://fileforums.com/showpost.php?...4&postcount=35

As for decompression error - it's been already reported and Razor answered.

And don't use Russian here, it's English-only forum.

There are internal errors I've come across, so for now until next update, don't use xtool. the expansion stops because of memory management issues, xtool keeps increasing memory usage over and over again until you run out of memory, I noticed this when I was running some tests.

As of right now, I have fixed this issue but am running more tests to make sure all is well.

FitGirl mentioned you must pick one codec in the zlib lineup, rzlib is what is best and it's the same one used in the old xtool, I just added the other ones (zlib and reflate) for special cases where a specific codec is required.

Update available

1908_R1
- Fixed MT hanging issues
- Fixed reflate issues
- Fixed memory issues
- Improved zlib codec
- Added auto reflate level detector
- Fixed aborting issues when used via stdio

Notes
If there is a file where this xtool fails to precompress better or the same as ztool, pzlib, reflate or precomp, please upload it.

Test xtool_1908_R1
 

Streams: ZLib
Middle-Earth - Shadow of Mordor ALL OK
Fallout 4 ALL OK

Wargame Red Dragon
Streams: Reflate
Log:

Can you test these exes on the same input, thanks

_binaries_fpc.7z Test
 

@Razor12911

x64
x86
But it works with t1,t2,t3,t4 in pre-compression \ t100p Works on decompression..
AMD RYZEN 5 1600 6-Core

1908_R1 x86 vs _binaries_fpc x86 - Test
 

1908_R1 x86 t100p for pre-compression = ERROR: write error (disk full?) in compression algorithm

1908_R1 x86
_binaries_fpc x86
__________________________________________________ __
The 1908_R1 x64 version works very stable and well.:D

Update available

1908_R2
- Updated history size feature
- Added save/load history file feature
- Improved processing speed

The benefits of the usage history database files

I'll use GTAV as an example, it's a huge game and has a lot of highly compressed streams.

So if you were compressing this game and you messed up the srep/lolz settings or your PC shutdown unexpectedly due to several reasons like powercuts or windows forcing updates and etc, if xtool already stored a database then it will load it up with all the information that was used in previous precompression session on the same input to speed up the process because it will know what needs to be done and what settings should be used.

Here's an example on update.rpf (GTAV)

on first run:
since I have processed this input before, xtool loaded up history file:
The reason I decided to add this feature was because of games compressed with ridiculously high compression settings resulting in precompression being very slow so then I wanted people to save time by allowing at least one of us to process the game then share their database file with everyone so that they can use it to save time on precompressing the same game.

This xtool is already faster than the old one and I just had to find more ways to give you guys more speed. :D

Note: It doesn't matter if the input is different from the one that was used to create a specific history file, an example to explain this, if one used GTAV with latest update to make a history file, a person who has the first release of GTAV without any updates can still use the same history file without problems or visa versa, if a game got updated and you wanted to compress it again with the new files, you can use the old history file.

Test 1908_R2
 

Game GTA V use x64w.rpf
Size : 893 MB


Creating archive

#1 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: rzlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.78 sec/real 91.65 sec = 2%. Speed 10.23 mB/s
All OK
-------------------------------------------------------------------------------------------------------

#2 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: zlib_GTAV.arc using rzlib
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.67 sec/real 107.49 sec = 2%. Speed 8.72 mB/s
All OK
-----------------------------------------------------------------------------------------------------------

#3reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Creating archive: reflate_GTAV.arc using reflate
Memory for compression 0b, decompression 0b, cache 16mb
Compressed 1 file, 937,340,928 => 937,341,048 bytes. Ratio 100.00%
Compression time: cpu 1.80 sec/real 61.87 sec = 3%. Speed 15.15 mB/s
All OK

---------------------------------------------------------------------------------------------------------------------------------

Extracting archive
#1 reflate

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: reflate_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.22 sec/real 27.71 sec = 12%. Speed 33.83 mB/s
All OK

-----------------------------------------------------------------------------------------------------------------------------------

#2 rzlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: rzlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.19 sec/real 59.93 sec = 5%. Speed 15.64 mB/s
All OK

-------------------------------------------------------------------------------------------------------------------------------------

#3 zlib

FreeArc 0.67 (March 15 2014) Using additional options: --logfile=_Compression.log
Extracting archive: zlib_GTAV.arc
Extracted 1 file, 937,341,048 => 937,340,928 bytes. Ratio 100.00%
Extraction time: cpu 3.06 sec/real 38.95 sec = 8%. Speed 24.07 mB/s
All OK

end :)

reflate didn't work with t100p when packing. add setting lvl. for reflate pls and add option for disable autodetect lvl. xD

There is much better way to detect cmp level than any current tools do(including precomp) and is simple to implement:

When you find first potential stream, first use all levels until match(as you probably do), but instead of fixing it for rest of data as granted, do at least say 9 more(up to 10). If 80%(or 8 of 10) of those streams show same specific level(say -9 or -6), then keep that for the rest of data.

If less than 80%, do all levels check for another 10 streams and average with previous 10. Now if average(of 20 streams) is 80% specific level then all is good for the rest of data, otherwise go full check again another 10 streams etc..

You can set minimum threshold to 70% if 80% threshold is too much but I think it should be fine. In other words, with this simple to implement algo you dont ever need option to manually set specific level, it will automatically determine whether streams are too mixed and it needs to test each on all levels(extremely unlikely), more likely it will eventually get right one that is in majority. This will also prevent false detection unless majority of first 10 streams are wrong levels, if thats common then you could do in steps of 100's instead of 10 and so on.

rly? lol. i'm look also on unpack time. example: better low lvl for save 1 min. my time or ~4 mb. better compression?

doesn't seem like libraries are loaded :)

Well up to you, if you know how to use reflate lvls

:eek::eek::eek: and you want me to remove auto detect.

This is why I add it in the first place, especially when it comes to reflate because most people don't know how it works. Low level does not mean you save time and at times, if you choose incorrect levels you end up with negative ratio. Example is GTAV, people used level 1 just because it was faster, resulted in horrible results then at some point people just decided to use level 9 on everything because "most" games are highly compressed, then game mad max which was level 1 compressed, they got shitty results again so yea, if you want level settings, I guess it's not a problem to add it but I added autodetect for a reason, because of people messing up

I'd like to see this method fly :D

especially with the case of xtool because of multi threading, precomp might find this useful because it is single threaded but xtool? no. It's not something I have not thought about before but look at it from this perspective. you're processing DiRT Rally, a game with large streams, so to detect level according to you is to process the first stream, check it level by level. Take note when it comes to deflate, there are not just 9 options (1-9), but there are 81 options, each level has memory settings (9 of them as well) with also affect the checksum of the stream if you are doing trial and error approach, so you're telling me it's best to try 81 levels on the first stream, so the next 9 streams can use the same method, during those 81 trial and errors, the other threads in xtool, what would the other threads be doing during that time? :D, a user selected to use 8 threads but it's still just 1 thread that needs to figure out what options to use and then after that is done, they can proceed. As those threads process the streams using the level they found from the first stream, if it's incorrect, they all now have to go trial and error for all those streams whichever way you look at it, this method has drawbacks in terms of time the other threads spend by not doing anything and just waiting, other drawback is, it's not really guaranteed that all those 81 options you'll be trying will give back perfect checksum so you can be making the other threads waiting for something that will never come.

xtool is fast not just because of multi threading, you can even compare it with precomp using 1 thread. I've thought about level detection and etc.

Here's how xtool works, the scanner itself can even give ideas of what level was used, zlib has a 2 byte header that is generated based on what level was used to compress a specific stream but sometimes this is not enough because some streams are headerless. This is where I decided to add a statistics system that stores information as all threads process the streams, this statistic system I introduced allows xtool to determine which cmp levels popped up the most and which ones were less, so if it were a case of trial and error, it wouldn't be trying from level 1-9, it would be the level that appeared the most using previous stream data(could even be in the order (6,5,9,8,1,2,3,4..). There are more strategies I've added for cmp level detection and they are all quick because they use stats

add setting - turn off only/

Razor,

you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?

As for my idea,
It don't need to go through all 81 every time, only until match is found and then next stream can be tested with that level found in previous stream first. This means if streams are not mixed there will be no slowdown because first try is a match already. Also your own test implementation on first stream seem fast, why not use same technique, whatever is it doing?

I have never noticed ztool/xtool being slow at beginning, it seems to start immediately. Would there really be such big difference if it had to try 10 streams instead of 1, especially with optimization I mentioned above(starting with previously found level)? Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?

As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long).

You mentioned using stats, that sounds actually very similar to what I am talking about. In fact, if I understand correctly you may have already implemented what I am talking about here :). But interestingly, you mentioned collecting stats from threads, that pretty much equals to cmp levels. So you in fact are collecting from multiple streams, not just first?

"you mentioned xtool does auto detection on the first stream. Does it iterate through all 81 levels until match is found, or only few known to be used most often?"
- it does for all streams, all 81 options but very quickly, takes milliseconds per stream to go through all 81 options.

"It don't need to go through all 81 every time"
- it doesn't go through all 81, sometimes even the first option it tries is correct and it stops, if it isn't correct then the 2nd one is most likely correct because of stats. XTool requests mode (value that appears most often), if that option fails, it requests mode in 2nd order (value that appears the 2nd most often) and so forth... which is why trials on all streams is necessary because it keeps the stats function filled with updated information about the current input.

"next stream can be tested with that level found in previous stream first."
- this method is the same as stat but it stores information limited to one stream which happens to be the previous stream, the approach of xtool is it stores a number of previous stream information, default is 256 streams.
Let's say for example
if the first stream was gave back level 9, the next stream is determined to be level 9, roughly the same as what you suggested, but what if the third stream is level 6? you have go through all the trials, ok fine, now you think then next stream is level 6 because of previous stream but what if that level 6 stream was just png image within the game data but instead the entire game was level 9? you try level 6 on the next stream it fails, then again you have to go through all 81 options again. XTool apporach thanks to stats, is level 9, level 9 and level 6 is what was stored, so for the next stream, it doesn't matter if the last one was level 6, it will try level 9 because it was detected twice compared to level 6, if it wasn't level 9, it will try that level 6 before it tries all 81 options because it was at least detected once, if that fails. it then tries 81 - 2 (6-9 from stats already failed) it does this for the past 256 streams and tries options in mode orders.

"I have never noticed ztool/xtool being slow at beginning, it seems to start immediately"
- It's because of stats plus other methods I've developed that make this possible.

"Perhaps up to few seconds of delay should not be much problem in scope of thousand streams and GB's of data?"
You'll notice not just "few seconds of delay", yes few seconds for a set of streams but minutes of overall delay :D if you were to use this method for big streams, try this on dirt rally or doom 2016 and you'll see some incredibly low CPU utilization when using a lot of threads because other threads are obliged to wait for the first stream.

"As for MT, that's not a big deal we don't want to test whole data only first couple of streams(again assuming it should not take long)."

with large streams, it is guaranteed to take long as mentioned before, which is why xtool faces too many problems with multi threading, I've instructed the threads to try 81 levels while using stats from all the streams they can find, as a result they end up fighting and crash each other :D it's hilarious but you get the idea, it's just best to try all levels on all streams but let stats decide in what order they should be presented in to speed up the process.

"You mentioned using stats, that sounds actually very similar to what I am talking about."
- Yes with the exception that not the previous stream option is considered but the previous 256 stream options are considered. If from those 256 options, there are 128 level 6s, 10 level 3s , 1 level 5 and 20 level 9s, it will try them in the order: 6 >> 9 >> 3 >> 5, then since all that failed, it begins trials, 1 - 9, it will skip, 3,5,6 and 9 because those failed, after that the new discovered level is added to stats, if it was 2, it will try the new order as 6 >> 9 >> 3 >> 5 >> 2, the reason all streams have to try all options is to get rid of non appearing options, example is when the data out of nowhere started to become level 4 out of nowhere, as stats are added, the number of level 9s will decrease because of these new levels 4s until they reach a certain threshold, then the order will now be 4 >> 6 >> 9 >> 3 >> 5 >> 2, as you can see level 4 over took level 6 but level 6 is still considered because before all of the level 4s there were a lot of level 6s so yes, I really did think deep about how to speed up xtool :)

"I understand correctly you may have already implemented what I am talking about here"
- Yep

"So you in fact are collecting from multiple streams, not just first?"
- Yes, I had to carefully do this else the threads crash themselves while fighting for the same stream or the same information, causes headaches but I find it funny when this happens, this is what causes most problems in pzlib/ztool and old xtool but if you properly implement it, you get to enjoy all the speed that comes with it.

Finally, now(!) I am in picture. Thanks for taking your time, it does make a difference when I understand internals.

Believe or not until now I thought (z)xtool simply scan first stream and.. that's it - use same info for *all* rest of data lol :). Reason I thought so was that game packs are likely using single specific level and that could in theory be enough(to pack the game, for which zxtools were designed).

Since that is not the case and your approach is clearly even much better than mine we can rest this case. Thanks again.

1908_R2_x86 > ERROR: write error (disk full?) in compression algorithm
1908_R2_x64 > good work

decode has a bug.

Even with t1 it uses all cores!

also with x64 you can get "ERROR: write error (disk full?) in compression algorithm"...
i tested on doom with reflate t100p with c384mb i get ( ERROR: write error (disk full?))...
i changed to t4 it work...

for the decode it work i tested with t1 and t100p, for t1 use 11% of the cpu and for t100p it use all core...
***it's so important to emphasis do not use t100p for decode if you don't have a good cpu cooler, better to use t50p...

Guys, I’ll repeat. If you encounter an error, use xtool without fa so it produces an exception message so I know what the problem is

reflate (XTool 1908_R2_x86)
 

#############################################
# reflate (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

reflate works well with c16m in x86.:rolleyes:

zlib (XTool 1908_R2_x86)
 

#############################################
# zlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

zlib is very unstable, once successful, failed several times:(

rzlib (XTool 1908_R2_x86)
 

#############################################
# rzlib (XTool 1908_R2_x86)
#############################################

Test File : bigfile.004.tiger 1.99 GB

rzlib always fails.:(

@Simorq

Looks like I'll have to go through debugging road once more :D

@everyone

Update available

1908_R3
- Compatiblity broken (old history db and inputs cannot work in this version)
- Added plugin support
- Added LZ4 codec placeholders (lz4,lz4hc)
- Added LZO codec placeholders (lzo1c,lzo1x,lzo2a)
- Added ZSTD codec placeholders (zstd)
- Added Oodle codec placeholders (lzna,kraken,mermaid,selkie,leviathan)
- Added LZX codec placeholders (lzx)
- rzlib codec removed (use zlib)
- Added x86 memory limit

Notes
Placeholder does not mean the codec is ready to be used universally, it's intended for plugins which can be found here

xtool_1908_R3 X64 ZLIB CODEC
eFootball Pes2020 Demo ... dt00_4K_x64.cpk
Zlib Codec is Stable .. i tested it Three Times ...:D
/////////////////////////////////////////////////////////////
xtool_1908_R3 X64 oodle Codec
Wolfenstein Youngblood ... chunkbase_9_pc.resources

oodle leviathan Stream Dont Work :o