Helping thread :)

[Bulat]

Originally Posted by FitGirl

The example string works just fine
srep+4x4:t4:b128m:lzma:128m

Never use large dict size for 4x4
4x4 means you're running 4 instances of LZMA, each with 400Mb dict size, that leads to 400*4*11 = 17600 MB of RAM to compress.

On user machine ISDone will require four continuous memory blocks of 400 MB each.

Also, using 4x4 hurts compression ratio due to blocking. That's why "some guy" repacks are larger than any other. You can try to use LZMA2 instead of LZMA1 (gives you 2 decompression threads by default), but it has it's own downsides.

I've personally used 4x4:LZMA in some of my early repacks, but gave it up. Number of SSD users is not that high, and unpacking in four streams on HDD often leads to overload.

i agree with almost everything you said except for two things:

1. 4x4 doesn't mean "use exactly 4 threads both for compression and decompression". let's see - data are split into independently compressed blocks, so you can use any number of threads at compression stage and, independent of that, any number of threads at decompression stage. both arc and unarc find largest contiguous block available (in 64-bit windows it will be 2047 MB), divide its size by the memory required for single (de)compression thread, and run so many threads. so, on 64-bit windows it will use more threads, on 32-bit windows it will use less threads

2. lzma2 is different from lzma in two things:
- it stores incompressible chunks of data
- like 4x4, it can split input data into independent blocks. unfortunately, it always uses only 1 thread for decompression. When manual says "LZMA2 uses: 1 thread for each chunk in x1 and x3 modes; and 2 threads for each chunk in x5, x7 and x9 modes" it means only comperssion stage

So, compared to 4x4:lzma, lzma2 may give slightly better ratio and single-thread speed, but it can't decompress even in 2 threads. but probably you meant compression w/o splitting into independent blocks, where lzma2 indeed is slightly better, but still single-threaded on decompression side. unfortunately, it's inherent property of lzma algorithm - you can't use morte than 1 thread to decompress single lzma compressed stream

3. that said, inability to decompress data compressed with 4x4:lzma is a bug in my program. I already have a better algorithm for InsertTempfile procedure, but ATM it's not yet ideal and still wrtitten in Lua. But i will fix that sooner or later.

4. these days, most of user computers run Win64, so probably it's time to switch to 64-bit unarc.dll in most repacks? i will try to build one.