FileForums - CLS-ZStd

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - CLS-ZStd (https://fileforums.com/showthread.php?t=98121)

Here's CLS version of ZStd, ran some tests, it's fast during compression because no temps are created, also added multi threading feature, it's only effective if you use a high level to compress, also if the chunk size is high as well, then it is effective, if you're using level 1 for example, and you try to use multi threading, just know that it is going to be slow, since level 1 is almost like copying a file, just adding multi threaded copying doesn't mean speed will be fast, it will drop.
During decompression, I ran tests as well, it's also fast, you can easily get more than 200mb/s. Decompression is single threaded.

This is nice i'll test.
Update.............
Nice speed.
I was tried chunks like 50m or 1g and 8b and the percent freeze in 9% or 100%. But in 1024k works very well. Maybe only accept k in chunks. Also pack and unpack no errors. Tested with 80 gb folder.
Threads 6
level 21
chunks 1024k

we need the felice2011 benchmark here!!!

Quote:

Originally Posted by ChronoCross (Post 450929)

Your PC freezes due to lack of RAM ;) ZStd is also an LZ-Compressor which uses about 10-20 times your Chunk size for better results. So, If ya enter 1 GB as chunk size, you'll need 20 GB RAM

Ok thanks for the tip.
With 8b the progress stack in 9% and complete the job without errors.
In 50m percent goes quick to 100% and stack to complete the job. In 1g arc.exe crash. In any case works this cls works faster.

1G chunks? Lol
And using 6 threads. That's 1GBx2x6 = 12GB Ram usage. X86 is limited to 2GB.
Memory usage is calculated as chunk size x 2 x number of threads

Hey wow ok. it's a test men. My tests goes from 0 to the limit of the program.

I place an quick test of compression and decompression with enwik8, with and without cls.dll, with a block of memory resident in the 16384k heap, to equal the same compression ratio in response to the use of zstd without the cls.

I also tried to use 8 threads, the speed of compression increases considerably, but the cpu ... squirts to hell, with freeze pc if not adequately cooled.

zstd without cls

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:18:24

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc          95,4°MB           26,2°MB       27,46 %              1°Files     00:00:42:60   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\

                                                                                                         -mzstd

==================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:20:16

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc   ...\zstd_exe.arc          26,2°MB       00:00:00:36              x -o+ -w.\  

==================================================================================================================================

zstd only cls

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:28:24

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc          95,4°MB           26,1°MB       27,36 %              1°Files     00:00:18:57   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\ 

                                                                                                         -mzstd

==================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:31:16

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc   ...\zstd_cls.arc          26,1°MB       00:00:00:43              x -o+ -w.\  

==================================================================================================================================

Code:

[CLS]

Level=22

ChunkSize=16384k

Threads=4

With the use of the cls the compression time decreases by almost 60% while maintaining the same compression ratio with and without cls.
On the contrary, the decompression time has a lower speed of 15-17% with the use of cls, but in this case I think you should test larger files.
These dll alternative by razor are a real boon from sky ...;)

Hello mates, I was not here from long time due to my studies actually, I managed some time today, so thought for a modified version by me in ZSTD :D

What I did?

1. Actually increased speed around atleast 2mb/s with the same cls through modified zstdlib_x86.dll without any ratio loss.

2. Added latest compiled version of Zstd x32 and x64, additionally PZstd also!

3. I did some benchmarks also, for all tests level 19 was used, was tested for compression only and threads were set as 2(It doesn't matter in PZstd case), I have maximum 2 threads only.

Test with new and older DLL

Code:



By using zstdlib_x86 Modified For Pzstd ---->>



Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%

Compression time: cpu 103.66 sec/real 59.58 sec = 174%. Speed 3.82 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 19:35:17.12



 Install Finished - 19:36:17.06





By using zstdlib_x86 Razor For Pzstd ---->>



Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%

Compression time: cpu 163.23 sec/real 98.28 sec = 166%. Speed 2.32 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 19:43:16.40



 Install Finished - 19:44:54.83





--------------------------------------------------------------------------------------------------------



By using zstdlib_x86 Razor For Zstd x32 ---->>



Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%

Compression time: cpu 160.95 sec/real 104.04 sec = 155%. Speed 2.19 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 19:23:26.29



 Install Finished - 19:25:10.48





By using zstdlib_x86 Razor For Zstd x64 ---->>



Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%

Compression time: cpu 166.00 sec/real 103.52 sec = 160%. Speed 2.20 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 19:26:41.28



 Install Finished - 19:28:24.95





-----------------------------------------------------------------------------------------------------------------------



By using zstdlib_x86 Modified For Zstd x32 ---->>



Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%

Compression time: cpu 109.63 sec/real 74.61 sec = 147%. Speed 3.05 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 19:30:54.96



 Install Finished - 19:32:09.99





By using zstdlib_x86 Modified For Zstd x64 ---->>



Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%

Compression time: cpu 106.92 sec/real 76.79 sec = 139%. Speed 2.97 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 19:33:06.21



 Install Finished - 19:34:23.35

Tests Between Zstd and Pzstd with the Modified DLL

Code:



Pzstd



By using Pzstd ---->>



Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%

Compression time: cpu 107.86 sec/real 65.95 sec = 164%. Speed 3.45 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 15:10:28.93



 Install Finished - 15:11:35.30







By using Zstd x32 ---->>



Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%

Compression time: cpu 105.44 sec/real 66.34 sec = 159%. Speed 3.43 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 15:21:16.69



 Install Finished - 15:22:24.00







By using Zstd x64 ---->>



Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%

Compression time: cpu 106.08 sec/real 67.47 sec = 157%. Speed 3.38 mB/s

All OK



*******************************************************************************



               Compressing is Completed



*******************************************************************************

 Install Started  - 15:23:24.16



 Install Finished - 15:24:32.64

Edit: I will prefer to use zstd32_gcc620.exe or zstd64_gcc620.exe as zstd.exe for now with current Razor CLS.

Quote:

Originally Posted by felice2011 (Post 450948)

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:18:24

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc          95,4°MB           26,2°MB       27,46 %              1°Files     00:00:42:60   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\

                                                                                                         -mzstd

==================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:20:16

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc   ...\zstd_exe.arc          26,2°MB       00:00:00:36              x -o+ -w.\  

==================================================================================================================================

zstd only cls

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:28:24

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc          95,4°MB           26,1°MB       27,36 %              1°Files     00:00:18:57   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\ 

                                                                                                         -mzstd

==================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:31:16

==================================================================================================================================

 ○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○



             Arc   ...\zstd_cls.arc          26,1°MB       00:00:00:43              x -o+ -w.\  

==================================================================================================================================

Code:

[CLS]

Level=22

ChunkSize=16384k

Threads=4

You can run bench again.

Quote:

Originally Posted by Razor12911 (Post 452564)

You can run bench again.

With the files posted by @MAK ?

Yup. I doubt that there will much diff though.

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:54:37

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        Arc+zstd          95,4°MB           26,2°MB       27,46 %              1°Files    00:00:44:940   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:56:59

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



Arc+zstd32_gcc620         95,4°MB           26,2°MB       27,46 %              1°Files    00:00:43:959   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:59:05

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



Arc+zstd64_gcc620         95,4°MB           26,2°MB       27,46 %              1°Files    00:00:43:956   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 14:09:55

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



Arc+pzstd_gcc620          95,4°MB           26,7°MB       27,99 %              1°Files    00:00:09:946   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mpzstd 

=============================================================================================================================================

All test ZSTD Same Options :

Code:

Level=19

ChunkSize=100M

Threads=2

Test PZSTD Options

Code:

pzstd -19 -f

Quote:

Originally Posted by felice2011 (Post 452585)

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:54:37

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



        Arc+zstd          95,4°MB           26,2°MB       27,46 %              1°Files    00:00:44:940   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:56:59

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



Arc+zstd32_gcc620         95,4°MB           26,2°MB       27,46 %              1°Files    00:00:43:959   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:59:05

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



Arc+zstd64_gcc620         95,4°MB           26,2°MB       27,46 %              1°Files    00:00:43:956   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================



-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 14:09:55

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



Arc+pzstd_gcc620          95,4°MB           26,7°MB       27,99 %              1°Files    00:00:09:946   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mpzstd 

=============================================================================================================================================

All test ZSTD Same Options :

Code:

Level=19

ChunkSize=100M

Threads=2

Test PZSTD Options

Code:

pzstd -19 -f

no matey, he actually mean, test between cls with zstd and cls without zstd for speed

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 17:14:41

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



       Arc+zstd          95,4°MB           26,2°MB       27,46 %              1°Files    00:00:43:973   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================

This ZSTD v1.0.0 normal NO-CLS, the previous post the tests already carried out with CLS.
same all settings "-19" performed with enwik8.
Try you also with the same input file:rolleyes:

Quote:

Originally Posted by felice2011 (Post 452588)

Code:

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 17:14:41

=============================================================================================================================================

 ○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○



       Arc+zstd          95,4°MB           26,2°MB       27,46 %              1°Files    00:00:43:973   a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd 

=============================================================================================================================================

This ZSTD v1.0.0 normal NO-CLS, the previous post the tests already carried out with CLS.
same all settings "-19" performed with enwik8.
Try you also with the same input file:rolleyes:

So, the conclusion in the last is, we finally made a zstd cls with no drawback of speed and ratio, and with full support to Inno, Am happy my zstdlib_x86.dll was useful here ;)