plzip_modded - parallel 64bit lzma (de)compressor and a direct 4x4:lzma replacement

[elit]

Plzip(not to be confused with pzlib) is a lzma compressor similar to xz, however fully scalable for multiple cores. Unlike other variants like 7zip/lzma2 or xz as well as many others, this one also fully support parallel decompression. It is therefore fully scalable similarly to "lolz -mt# -mtt1" but with pure lzma and therefore quicker. It is much better option IMO than 7z or xz as a replacement to FreeArc's internal (4x4)lzma. Plzip fully support <stdio> both ways and for both compression and decompression. Therefore I consider it best 64bit alternative to current 4x4:lzma available - as of now.

However, default plzip intentionally hinder itself on decompression by limiting memory for thread's splitter/muxer, therefore by default any bigger number of cores and bigger dictionaries result in using again only 1/2 cores during decompression, putting it basically back to xz/lzma2/7z level. That despite potential and design it pose. Or to be more precise, it does utilize all the core available, but only if you decompress from physical file to another, <stdio> will simply scale back. I had a mail exchange with plzip's original author about this where he confirmed this design.

Which is where my mod comes. As FA's (ahead of its time) 4x4 can prove, things can be done with reasonable memory management and better. This mod change memory limits for decompression(for splitter/muxer not lzma's).
Specifically, with this mod you can use block size up to 128m and up to 16 threads during compression and still decompress utilizing full 16t cpu load. You can of course use bigger block size and even more threads as much as default software support, but then decompression will start scaling back again. But I think this is enough for a foreseeable time. Furthermore and just like lolz, you can compress today with different number of threads and in the future when maybe 16t cpu will become normal, it will utilize it already without need for re-compression. Thats why -n parameter for decompression as well.

Finally, I also modded lzma's lc parameter from default 3 to 8, making it equivalent to lzma:lc8 of FA. But, you can now also only decompress with this modded plzip only, because its hardcoded at compile time. Memore utilization for decompressing (minus for lzma * threads) should be around up to ~2Gb. I have tested it on 4t cpu with 128m block size and it loaded all cores upon decompression as expected. Original plzip utilized less memory, but was slower and only used ~1t with that setting. This mod should now be similar in behavior to 4x4, which you dont need anymore but if you want there is also simple minilzip exe which is single threaded 64bit lzma. That one you can use with 4x4 like: "4x4:b64m:t#:minilzip:s32Mi:...". Yes 4x4 support more than 4 cores ^_^. But you need block size of 4x4 2x of dictionary size of external packer or it will crash.

Here is arc.ini example:

[External compressor:plzip]
default = -s 64Mi -m 32 -B 128Mi -n 4
packcmd   = plzip.exe -c {options} - - <stdin> <stdout>
unpackcmd = plzip.exe -c -d -n 4   - - <stdin> <stdout> 

[External compressor:lzlib]
default = -s 64Mi -m 32 
packcmd   = minilzip.exe -c {options} - - <stdin> <stdout>
unpackcmd = minilzip.exe -c -d        - - <stdin> <stdout>

I would like to make more of it in future but dont have time yet, like adding command line options for more basic lzma settings such as mc and lc. Also 4x4 still have one *huge* advantage, it can skip uncompresseable data chunks, that was(and still is) one thing keeping it ahead of lower quality(yes! deal with it!) variants like 7z and pretty much everything else out there. I want that feature badly, but have no idea how to code it. Supposedly 4x4 use something called "order 0" fast compression which if it compress chunk to worse than 99% ratio will just copy over instead of handling to lzma. I may ask at encode about this as I plan to up my mod there as well. That chunk skipping give around ~50%-300% speed boost according to my tests and is independent extra thing from $compressed -ms, like I said way ahead of its time.

Anyway, the mod(based on plzip 1.7 and lzlib 1.9):
EDIT: New re-upped file include bad entropy skipping and source code to be in sync with original open source license.
plzip-1.7_mod2_pre.7z

PS(test for me decompression with above arc.ini if you have 16 core cpu with 128Mi block size archive and big file to confirm it really load 100% cpu sustained)

[doofoo24]

will it work with Edison007 isdone mod without exe ?

[elit]

This is a program not cls, you need also its libraries, if that what you meant.

[Gupta]

nice,
i will suggest you to use msys2, compile with -static -s and you don't have to carry dll with you

[elit]

Originally Posted by Gupta

nice,
i will suggest you to use msys2, compile with -static -s and you don't have to carry dll with you

For pure windows it require patch to translate pread/pwrite which gnu on windows lack(without cygwin). But, patch already exist and I have it, its just that I didnt bother working it in yet. But speed wise its same dont worry, I compared with one published at encode which is pure mingw. I will probably apply it down the road when I modify compressor further and update post again. For now dont let few extra dll's bother you .

[panker1992]

7zip 18.03 that was released, now supports parallel decompression on lzma2/xz
and also it supports up to 40% more speed on single thread

except that how plzip fares on ratio ?

[elit]

Originally Posted by panker1992

7zip 18.03 that was released, now supports parallel decompression on lzma2/xz
and also it supports up to 40% more speed on single thread

except that how plzip fares on ratio ?

Really? I have to check, if true its a progress finally, although still surely lack bad entropy skipping techniques like 4x4 have. Anyway when I tested against FA 4x4:xlzma It did compress a tiny bit better(~0.5%), probably because of missing entropy skipping. When binded to 4x4 as 4x minilzip it was almost same speed with again tiny better ratio. Like maybe 6s slower(of 1:40min) and 20-60mb less compressed size vs xlzma(of 2-3gb original). Btw without 4x4's cool entropy skipping compression time was 4-5min vs 1:40min, in line with xz and 7z vs 4x4 and 2-3x slower just like them. So you see the difference, now imagine compressing 50gb+ data...

In nutshell they are all pretty much same lzma's to me, its the smart techniques that make differences.

[panker1992]

I know how it works

me and Razor made a custom reflate tool looooong time ago at that time we didnt add entropic search inside !!

(we also found that entropy that is above 5.5 signify compressed ~tiny packed ~ data)

after that Razor took that code and made ZTOOL, ztool has that you seek inside which can bypass data that arent compatible with the program's parameters

you can ask for his help!! i don't have the code to help you do it surely Bulat is ahead of his time ... hehehe

[elit]

Thank you I do have entropy code already from Bulat, which in turn is from Matt's zpaq. Bulat also pointed at his own src code so there is no deficit anymore. Its only time that I need plus finishing many other things.

But to your question, I see plzip on par with 7z/xz when it come to compression, speed, quality and stability. Currently it doesn't expose all lzma params to user. But its absolutely simple low count code is great for modification. Its still more scalable I think as lzma2 probably doesnt support >8 cores whereas this is made from ground up design to any number. It also incorporate lzma1 which support lc8(lc>4) unlike lzma2. All in all this is perfect candidate for NxN:lzma and usable/stable already now if you don't need all lzma params or entropy skipping.

[panker1992]

I haven't tested your modded tool yet but i plan to

if you can digest that 99> entropy to skip data and make it somehow comparable to 4x4 but with ratio of 7z/xz (lzma2) then this tool has a shot !!

Btw you can optimize 7z/xz to as many threads as you want but yours will have skipping which will make it faster by default

Ex:

[External compressor:lzma2]
; Block Size = -ms256m | Multithread = -mmt=8 |
header = 0
packcmd = 7z a -txz -an -m0=lzma2{ption}:fb=273:mf=bt4:mc=100000000:lc=4: lp=0 -mmt=4 -ms1024m -mx9 -si -so <stdin> <stdout>
unpackcmd = 7z x -txz -an -y -si -so <stdin> <stdout>

[External compressor:lzma2_alt]
; Alternative Chunking Method for multithreading decompression
header = 0
packcmd = 7z a -txz -an -m0=lzma2:d512m:c1024m:fb=273:mf=bt4:mc=100000000:l c=4:lp=0 -mmt=4 -mx9 -si -so <stdin> <stdout>
unpackcmd = 7z x -txz -an -y -si -so <stdin> <stdout>

[elit]

entropy.png

^_^ ,but wait a bit longer I want to finish this properly and add extra options for user(some lzma options like mc (& lc), memory limit for decompression slots option etc).

PS(mods, I am not sure if this post belong to conversion tutorials I think I made mistake, pls feel free to move it if you think its appropriate).

[panker1992]

Originally Posted by elit

Attachment 21442

^_^ ,but wait a bit longer I want to finish this properly and add extra options for user(some lzma options like mc (& lc), memory limit for decompression slots option etc).

PS(mods, I am not sure if this post belong to conversion tutorials I think I made mistake, pls feel free to move it if you think its appropriate).

i know you want to do it properly

maybe if you want you can add some filters, or maybe content modelling preferably sound for example lolz by profragger has textures modelling

[elit]

Originally Posted by panker1992

i know you want to do it properly

maybe if you want you can add some filters, or maybe content modelling preferably sound for example lolz by profragger has textures modelling

This is final stage, general purpose lzma compressor designed mainly for FreeArc and operating on full data blocks. You should use filters from within FA and appropriate compressors for each group(There is already TTA, oggre etc for sound files).

However, what I can do in future is to create variants, for example how would you like fully parallel Razor compressor that also support stdio, instead of lzma? I bet people would get crazy over it. But for that I would need support from its original author to access functions as it is closed source. Was also thinking lzss if its better or lzham and so on. I can inject different compressors but certainly not filters as this is not feasible with its current design.

But I can say already this plzip is very solid(as a lzma). With all other options and better user control it will be better than either xz or latest 7-zip. If I can implement it as I think that is. It also support lc8, unlike those.

[panker1992]

razor cannot be multithreaded as it is right now due to the fact that it doesnt create blocks, its near optimal parsing cannot allow for it (author himself explained)

as to stdio i was the one to ask Christian to make it and he said "no" because he goes for Unicode!

i like your tool and will try it

like your ideas and really wanna see how your modded stuff can be unique in a sense

[elit]

Here is with finished bad entropy data skipping(raw copying), set at 97.9% currently. Anything above is copy. Raw data are still CRC checked. This part is considered stable by me now so let me know if you find anything.

From now on I will focus on changing/adding options. So next release should be completely new named compressor in new thread(I will refer link here). I dont know exactly how long it will take, maybe few days or weeks depending on how things go and my time. Btw arc.ini is like this:

[External compressor:plzip]
default = -s 64Mi -m 32 -B 128Mi -n 4
packcmd   = plzip.exe -c {options} - - <stdin> <stdout>
unpackcmd = plzip.exe -c -d -n 4   - - <stdin> <stdout>

Newly named compressor will most likely be called xnlz, as for 4x4:lzma -> NxN:lzma -> xnlzma -> xnlz. Also because it will have extra header flags it will not be compatible with this so keep that in mind if you want to use it already. Also neither this already nor xnlz are/will be compatible with plzip.

EDIT: vis first post