plzip_modded - parallel 64bit lzma (de)compressor and a direct 4x4:lzma replacement

[elit]

Plzip(not to be confused with pzlib) is a lzma compressor similar to xz, however fully scalable for multiple cores. Unlike other variants like 7zip/lzma2 or xz as well as many others, this one also fully support parallel decompression. It is therefore fully scalable similarly to "lolz -mt# -mtt1" but with pure lzma and therefore quicker. It is much better option IMO than 7z or xz as a replacement to FreeArc's internal (4x4)lzma. Plzip fully support <stdio> both ways and for both compression and decompression. Therefore I consider it best 64bit alternative to current 4x4:lzma available - as of now.

However, default plzip intentionally hinder itself on decompression by limiting memory for thread's splitter/muxer, therefore by default any bigger number of cores and bigger dictionaries result in using again only 1/2 cores during decompression, putting it basically back to xz/lzma2/7z level. That despite potential and design it pose. Or to be more precise, it does utilize all the core available, but only if you decompress from physical file to another, <stdio> will simply scale back. I had a mail exchange with plzip's original author about this where he confirmed this design.

Which is where my mod comes. As FA's (ahead of its time) 4x4 can prove, things can be done with reasonable memory management and better. This mod change memory limits for decompression(for splitter/muxer not lzma's).
Specifically, with this mod you can use block size up to 128m and up to 16 threads during compression and still decompress utilizing full 16t cpu load. You can of course use bigger block size and even more threads as much as default software support, but then decompression will start scaling back again. But I think this is enough for a foreseeable time. Furthermore and just like lolz, you can compress today with different number of threads and in the future when maybe 16t cpu will become normal, it will utilize it already without need for re-compression. Thats why -n parameter for decompression as well.

Finally, I also modded lzma's lc parameter from default 3 to 8, making it equivalent to lzma:lc8 of FA. But, you can now also only decompress with this modded plzip only, because its hardcoded at compile time. Memore utilization for decompressing (minus for lzma * threads) should be around up to ~2Gb. I have tested it on 4t cpu with 128m block size and it loaded all cores upon decompression as expected. Original plzip utilized less memory, but was slower and only used ~1t with that setting. This mod should now be similar in behavior to 4x4, which you dont need anymore but if you want there is also simple minilzip exe which is single threaded 64bit lzma. That one you can use with 4x4 like: "4x4:b64m:t#:minilzip:s32Mi:...". Yes 4x4 support more than 4 cores ^_^. But you need block size of 4x4 2x of dictionary size of external packer or it will crash.

Here is arc.ini example:

[External compressor:plzip]
default = -s 64Mi -m 32 -B 128Mi -n 4
packcmd   = plzip.exe -c {options} - - <stdin> <stdout>
unpackcmd = plzip.exe -c -d -n 4   - - <stdin> <stdout> 

[External compressor:lzlib]
default = -s 64Mi -m 32 
packcmd   = minilzip.exe -c {options} - - <stdin> <stdout>
unpackcmd = minilzip.exe -c -d        - - <stdin> <stdout>

I would like to make more of it in future but dont have time yet, like adding command line options for more basic lzma settings such as mc and lc. Also 4x4 still have one *huge* advantage, it can skip uncompresseable data chunks, that was(and still is) one thing keeping it ahead of lower quality(yes! deal with it!) variants like 7z and pretty much everything else out there. I want that feature badly, but have no idea how to code it. Supposedly 4x4 use something called "order 0" fast compression which if it compress chunk to worse than 99% ratio will just copy over instead of handling to lzma. I may ask at encode about this as I plan to up my mod there as well. That chunk skipping give around ~50%-300% speed boost according to my tests and is independent extra thing from $compressed -ms, like I said way ahead of its time.

Anyway, the mod(based on plzip 1.7 and lzlib 1.9):
EDIT: New re-upped file include bad entropy skipping and source code to be in sync with original open source license.
plzip-1.7_mod2_pre.7z

PS(test for me decompression with above arc.ini if you have 16 core cpu with 128Mi block size archive and big file to confirm it really load 100% cpu sustained)