Update available

Changes

- fixed command line parser bug
- updated library support

Hi thanks for the great work.

With xtool0.12 I have

On a testing file set mostly zip and 7z files, it could precomp it well
Then I could add srep+FL2 to compress it with a good ratio.

With the recent release xtool 2020 from xtool_0.3.8.7z,

with a setting of

It will not do the similar precomp process, output is

Do I have to adjust/add the parameter to have the similar results?

Edit:

I guess the default values for zlib might have changed. And on some aspects the new version is performing much more like a normal average setting.


For a set of .dat files 800MB, tried to compressing with arc

srep+FL2 compressed it to 414300KB, takes 32 seconds.

xtool_0.12 e:precomp:t75p,c128m:zlib+srep+FL2 compressed to 412000KB takes a big 55 seconds. the zlib codec here is working very hard on something and found a little bit to more to compress further. But it added 23 seconds up on (srep+FL2)

xtool_0.12 e:precomp:t75p,c128m:zstd+srep+FL2 got 414000KB takes 36 seconds, it adds 4 seconds up on (srep+FL2)

xtool_0.3.8 e:precomp:t75p,c128m:zlib+srep+FL2 got to 414000KB takes 33 seconds, it adds 1 seconds up on (srep+FL2)

xtool_0.3.8 e:precomp:t75p,c128m:zstd+srep+FL2 got to 414000KB takes 33 seconds , it adds 1 seconds up on (srep+FL2)

Update available

Changes

- fixed future stream bug

@github

The command line has changed, check the example to see how it's used.

Furthermore, the old xtool used reflate without the knowledge of the user. The new one doesn't so if you have a scenario where the old xtool gives better results, then you may need to combine zlib with reflate or preflate.

Wdl
 

small test on WDL

Thank you. So with xtool 2020 I should use zlib+reflate -d or zlib+preflate -d
on command line with this it does detect many streams

how to I pass the -d level in the arc.ini?
does xtool0.12/2020 support the writing of {options} in arc.ini?

I tried

Those doesn't make any difference to the one without d3 option

are you precompressing something that requires depth level to be set to 3 or higher? as for choosing between reflate or preflate, you need to know how each works as they have their advantages and disadvantages.

which should explain their differences seen here. What I would point out as the main difference, reflate supports streams with no tail while preflate doesn't so if a special deflate encoder was used or if a stream had been cut off, preflate will be unable to process such streams but reflate is able to handle these.

as for {option} via freearc, I'd first say you need to understand the command line syntax of xtool. The old xtool and new xtool use different syntax.

Ideally this would become

then used as -mxtool:mzlib:c32mb:t100p from freearc.

Thank you for the example.

I have it setup up like this


However both setups can not to pass in like xtool:mzlib+zstd since + is parsed by freearc, but I guess it's rarely needed to pass in to methods.

I asked about passing in the -d option, since there are up to 10, just saying to test it so I used 3 or 5, and in my case I do need to use -d3 or -d5 to get the same result as xtool 0.12

This true.

I wonder though, A depth of 3-5. I asked before if there is such data you have on hand because this is basically the same as having data, compress it using zlib, the output, compress that using zlib, do this at least 5 times. That's what depth is.

A depth of 1-2 is usually enough on everyday files such as pictures and documents as these are usually shipped in a zip file, that zip file is compressed using deflate and if it contains pdfs or png images then, that's when you need to use depth otherwise, don't touch it as you'll be increasing precompression time unnecessarily and have no gains in ratio.

Hello Razor, is there anything that can be done to cap the memory usage of xtool at a certain amount?

I am aware of a cmem variable in arc.ini packcmd (only small knowledge) however I was wondering if there's any other ways about this.

I find on larger workloads/workloads with high amounts of streams, xtool fills up my RAM and completely crashes my PC.

I am working with 16GB memory. Is there anything that can be done without changing chunk size?

Currently I'm capping threads and using $$arc$$ instead of stdio.

-lm

:D:D thank you for implementing such a feature, I thought it got removed when the project was started (I remember it being in 2019 version).

Thanks again.

Hello Razor,Masquerade is there anything that can be done to cap the memory usage of xtool at a certain amount?
Can you give an example of how we will apply this method

you can't cap the memory usage of the program, you can only reduce it. When you are compressing, usually each thread gets its own chunk. Using -lm parameter makes all thread use 1 chunk. As for memory usage, that's highly dependent on the user and the data they are precompressing, if you set very high chunk sizes, be prepared for high memory usage. When decompressing, xtool tries to not use more than 512 MB ram automatically unless if there are large streams or if the recompressing library requires more memory.

did quick test on mad max with/out --dedup=xtool.bin to se mem usage...
with --dedup size to 55gb
without --dedup size 57.1gb
but memory dec the same around 6gb.
*note only test with mzlib+srep

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mzlib -c128mb -t12 --dbase --dedup=xtool.bin - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>


[External compressor:srep]
header = 0
packcmd = srep.exe -m3f -d1g -a2 $$arcdatafile$$.tmp - <stdout>

Mad Max is a bad example for dedup testing. They use really huge deflate chunks with level 1 compression, so the amount of dupe chunks is not very big, unlike the duplicated data in those chunks when decompressed, that's why srep is good solution here.