CLS-ZStd

[Razor12911]

Here's CLS version of ZStd, ran some tests, it's fast during compression because no temps are created, also added multi threading feature, it's only effective if you use a high level to compress, also if the chunk size is high as well, then it is effective, if you're using level 1 for example, and you try to use multi threading, just know that it is going to be slow, since level 1 is almost like copying a file, just adding multi threaded copying doesn't mean speed will be fast, it will drop.
During decompression, I ran tests as well, it's also fast, you can easily get more than 200mb/s. Decompression is single threaded.

[ChronoCross]

This is nice i'll test.
Update.............
Nice speed.
I was tried chunks like 50m or 1g and 8b and the percent freeze in 9% or 100%. But in 1024k works very well. Maybe only accept k in chunks. Also pack and unpack no errors. Tested with 80 gb folder.
Threads 6
level 21
chunks 1024k

we need the felice2011 benchmark here!!!

[RamiroCruzo]

This is nice i'll test.
Update.............
Nice speed.
I was tried chunks like 50m or 1g and 8b and the percent freeze in 9% or 100%. But in 1024k works very well. Maybe only accept k in chunks. Also pack and unpack no errors. Tested with 80 gb folder.
Threads 6
level 21
chunks 1024k

we need the felice2011 benchmark here!!!

Your PC freezes due to lack of RAM ;) ZStd is also an LZ-Compressor which uses about 10-20 times your Chunk size for better results. So, If ya enter 1 GB as chunk size, you'll need 20 GB RAM

[ChronoCross]

Ok thanks for the tip.
With 8b the progress stack in 9% and complete the job without errors.
In 50m percent goes quick to 100% and stack to complete the job. In 1g arc.exe crash. In any case works this cls works faster.

[Razor12911]

1G chunks? Lol
And using 6 threads. That's 1GBx2x6 = 12GB Ram usage. X86 is limited to 2GB.
Memory usage is calculated as chunk size x 2 x number of threads

[ChronoCross]

Hey wow ok. it's a test men. My tests goes from 0 to the limit of the program.

[felice2011]

I place an quick test of compression and decompression with enwik8, with and without cls.dll, with a block of memory resident in the 16384k heap, to equal the same compression ratio in response to the use of zstd without the cls.

I also tried to use 8 threads, the speed of compression increases considerably, but the cpu ... squirts to hell, with freeze pc if not adequately cooled.

zstd without cls


-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:18:24
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:42:60 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\
-mzstd
================================================== ================================================== ==============================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:20:16
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc ...\zstd_exe.arc 26,2°MB 00:00:00:36 x -o+ -w.\
================================================== ================================================== ==============================


zstd only cls


-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:28:24
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc 95,4°MB 26,1°MB 27,36 % 1°Files 00:00:18:57 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\
-mzstd
================================================== ================================================== ==============================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:31:16
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc ...\zstd_cls.arc 26,1°MB 00:00:00:43 x -o+ -w.\
================================================== ================================================== ==============================



[CLS]
Level=22
ChunkSize=16384k
Threads=4


With the use of the cls the compression time decreases by almost 60% while maintaining the same compression ratio with and without cls.
On the contrary, the decompression time has a lower speed of 15-17% with the use of cls, but in this case I think you should test larger files.
These dll alternative by razor are a real boon from sky ...;)

[~MAK~]

Hello mates, I was not here from long time due to my studies actually, I managed some time today, so thought for a modified version by me in ZSTD :D

What I did?

1. Actually increased speed around atleast 2mb/s with the same cls through modified zstdlib_x86.dll without any ratio loss.

2. Added latest compiled version of Zstd x32 and x64, additionally PZstd also!

3. I did some benchmarks also, for all tests level 19 was used, was tested for compression only and threads were set as 2(It doesn't matter in PZstd case), I have maximum 2 threads only.


Test with new and older DLL


By using zstdlib_x86 Modified For Pzstd ---->>

Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 103.66 sec/real 59.58 sec = 174%. Speed 3.82 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 19:35:17.12

Install Finished - 19:36:17.06


By using zstdlib_x86 Razor For Pzstd ---->>

Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%
Compression time: cpu 163.23 sec/real 98.28 sec = 166%. Speed 2.32 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 19:43:16.40

Install Finished - 19:44:54.83


--------------------------------------------------------------------------------------------------------

By using zstdlib_x86 Razor For Zstd x32 ---->>

Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%
Compression time: cpu 160.95 sec/real 104.04 sec = 155%. Speed 2.19 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 19:23:26.29

Install Finished - 19:25:10.48


By using zstdlib_x86 Razor For Zstd x64 ---->>

Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%
Compression time: cpu 166.00 sec/real 103.52 sec = 160%. Speed 2.20 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 19:26:41.28

Install Finished - 19:28:24.95


-----------------------------------------------------------------------------------------------------------------------

By using zstdlib_x86 Modified For Zstd x32 ---->>

Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 109.63 sec/real 74.61 sec = 147%. Speed 3.05 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 19:30:54.96

Install Finished - 19:32:09.99


By using zstdlib_x86 Modified For Zstd x64 ---->>

Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 106.92 sec/real 76.79 sec = 139%. Speed 2.97 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 19:33:06.21

Install Finished - 19:34:23.35




Tests Between Zstd and Pzstd with the Modified DLL



Pzstd

By using Pzstd ---->>

Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 107.86 sec/real 65.95 sec = 164%. Speed 3.45 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 15:10:28.93

Install Finished - 15:11:35.30



By using Zstd x32 ---->>

Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 105.44 sec/real 66.34 sec = 159%. Speed 3.43 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 15:21:16.69

Install Finished - 15:22:24.00



By using Zstd x64 ---->>

Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 106.08 sec/real 67.47 sec = 157%. Speed 3.38 mB/s
All OK

************************************************** *****************************

Compressing is Completed

************************************************** *****************************
Install Started - 15:23:24.16

Install Finished - 15:24:32.64




Edit: I will prefer to use zstd32_gcc620.exe or zstd64_gcc620.exe as zstd.exe for now with current Razor CLS.

[Razor12911]

I place an quick test of compression and decompression with enwik8, with and without cls.dll, with a block of memory resident in the 16384k heap, to equal the same compression ratio in response to the use of zstd without the cls.

I also tried to use 8 threads, the speed of compression increases considerably, but the cpu ... squirts to hell, with freeze pc if not adequately cooled.

zstd without cls


-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:18:24
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:42:60 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\
-mzstd
================================================== ================================================== ==============================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:20:16
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc ...\zstd_exe.arc 26,2°MB 00:00:00:36 x -o+ -w.\
================================================== ================================================== ==============================


zstd only cls


-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:28:24
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc 95,4°MB 26,1°MB 27,36 % 1°Files 00:00:18:57 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\
-mzstd
================================================== ================================================== ==============================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:31:16
================================================== ================================================== ==============================
○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○

Arc ...\zstd_cls.arc 26,1°MB 00:00:00:43 x -o+ -w.\
================================================== ================================================== ==============================



[CLS]
Level=22
ChunkSize=16384k
Threads=4


With the use of the cls the compression time decreases by almost 60% while maintaining the same compression ratio with and without cls.
On the contrary, the decompression time has a lower speed of 15-17% with the use of cls, but in this case I think you should test larger files.
These dll alternative by razor are a real boon from sky ...;)

You can run bench again.

[felice2011]

You can run bench again.

With the files posted by @MAK ?

[Razor12911]

Yup. I doubt that there will much diff though.

[felice2011]

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:54:37
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:44:940 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:56:59
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd32_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:959 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:59:05
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd64_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:956 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 14:09:55
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+pzstd_gcc620 95,4°MB 26,7°MB 27,99 % 1°Files 00:00:09:946 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mpzstd
================================================== ================================================== =========================================


All test ZSTD Same Options :

Level=19
ChunkSize=100M
Threads=2


Test PZSTD Options

pzstd -19 -f

[~MAK~]

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:54:37
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:44:940 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:56:59
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd32_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:959 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:59:05
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd64_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:956 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 14:09:55
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+pzstd_gcc620 95,4°MB 26,7°MB 27,99 % 1°Files 00:00:09:946 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mpzstd
================================================== ================================================== =========================================


All test ZSTD Same Options :

Level=19
ChunkSize=100M
Threads=2


Test PZSTD Options

pzstd -19 -f


no matey, he actually mean, test between cls with zstd and cls without zstd for speed

[felice2011]

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 17:14:41
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:973 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================


This ZSTD v1.0.0 normal NO-CLS, the previous post the tests already carried out with CLS.
same all settings "-19" performed with enwik8.
Try you also with the same input file:rolleyes:

[~MAK~]

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 17:14:41
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:973 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================


This ZSTD v1.0.0 normal NO-CLS, the previous post the tests already carried out with CLS.
same all settings "-19" performed with enwik8.
Try you also with the same input file:rolleyes:

So, the conclusion in the last is, we finally made a zstd cls with no drawback of speed and ratio, and with full support to Inno, Am happy my zstdlib_x86.dll was useful here ;)

[Razor12911]

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:54:37
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:44:940 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:56:59
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd32_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:959 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:59:05
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd64_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:956 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 14:09:55
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+pzstd_gcc620 95,4°MB 26,7°MB 27,99 % 1°Files 00:00:09:946 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mpzstd
================================================== ================================================== =========================================


All test ZSTD Same Options :

Level=19
ChunkSize=100M
Threads=2


Test PZSTD Options

pzstd -19 -f

felice, chunk size is set to 100m, input is less than 100m, obviously no comparison with pzstd here because cls-zstd was single threaded either way.

Chunk size works this way.

It feeds a specified chunk size to a thread.
If you have 4 threads for example, you set 100m chunk size on a 360mb input.
thread 1-3 will get 100mb
thread 4 will get 60mb
for your case, it really doesn't matter how many threads you set, compression will always be single threaded because chunk size is bigger than main input, all this just goes to 1 thread, all other threads have nothing to process.

[felice2011]

Excellent elucidation, when I have time I'll just repeat the test..;)..though honestly I think the difference will be minimal, but I hope to be wrong.

[felice2011]

Today, having a little free time I repeated the test...:)...All Test Input Data 420MB.

Here we have as a benchmark compression via ZSTD v1.0 no CLS.

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 09:58:52
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd_1.0 419,6°MB 195,9°MB 46,69 % 14°Files 00:03:22:986 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================



Here the version tested ZSTD540 with CLS, with variables set "MT = 2 or 4" and chunksize in a range from 50m to 500m.

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:31:18
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt2cs50 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:946 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:32:47
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt4cs50 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:932 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:34:25
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt2cs64 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:948 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:35:18
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt4cs64 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:951 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:15:35
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt2cs100 419,6°MB 199,2°MB 47,47 % 14°Files 00:01:35:918 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:22:12
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt4cs100 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:942 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:23:41
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt2cs250 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:944 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:26:36
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt4cs250 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:937 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:28:34
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt2cs500 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:951 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:29:56
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd540mt4cs500 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:941 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================


Here the version tested zstd620 with CLS, with the same variables set "MT = 2 or 4" and the chunksize in a range from 50m to 512m.

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:38:55
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt2cs50 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:947 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:41:04
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt4cs50 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:932 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:42:46
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt2cs64 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:944 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:43:55
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt4cs64 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:944 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:45:15
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt2cs128 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:950 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:46:23
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt4cs128 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:03:04 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:47:33
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt2cs256 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:941 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:48:40
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt4cs256 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:957 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:49:40
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt2cs512 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:944 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 12:50:02
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+zstd620mt4cs512 419,6°MB 235,6°MB 56,15 % 14°Files 00:00:02:948 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
================================================== ================================================== =========================================




As you can see the difference between the versions ZSTD 540 & 620 with CLS is nothing in percentage of compression, and of a few thousandths of seconds in compression time.
A distinct difference between the ZSTD version v1.0, and versions with no CLS 540 and CLS 620, with the latter results in a marked improvement in speed at the expense of better compression with ZSTD v1.0, so the choice is purely subjective, whether to use the version with or without CLS, but personally have not seen anything of sensational in these versions with CLS support.
Razor your lesson on the block of memory and the allocation thereof is correct, but in this case will be interpreted theoretically, in practice way I did not have any difference of advantage or disadvantage, both as a percentage of compression, that compression time , whatever of the range utilized for the memory blocks.

[doofoo24]

is there update for cls-zstd to the latest ver 1.2 ?

[Razor12911]

is there update for cls-zstd to the latest ver 1.2 ?

I'm pretty sure dropping in a v1.2 dll wouldn't be much of a problem. ;)