|
#1
19-08-2016, 08:33
|
||||
|
||||
|
CLS-ZStd
Here's CLS version of ZStd, ran some tests, it's fast during compression because no temps are created, also added multi threading feature, it's only effective if you use a high level to compress, also if the chunk size is high as well, then it is effective, if you're using level 1 for example, and you try to use multi threading, just know that it is going to be slow, since level 1 is almost like copying a file, just adding multi threaded copying doesn't mean speed will be fast, it will drop.
During decompression, I ran tests as well, it's also fast, you can easily get more than 200mb/s. Decompression is single threaded. 
|
| The Following 22 Users Say Thank You to Razor12911 For This Useful Post: | ||
-XCX- (30-12-2016), 0xffaded (30-06-2017), Amsal (19-08-2016), Andu21 (05-10-2016), arkantos7 (20-08-2016), COPyCAT (31-08-2016), coveiro (19-08-2016), gozarck (19-08-2016), Gupta (19-08-2016), hasandgn34 (21-08-2016), JRD! (07-10-2016), kassane (20-08-2016), kenzo34 (19-08-2016), MCWRX (09-10-2016), Mini (20-08-2016), minivj (19-08-2016), NexusIV (20-08-2016), Prince4 (20-08-2016), RamiroCruzo (19-08-2016), SAM2712 (24-08-2016), Simorq (19-08-2016), ~MAK~ (07-10-2016) | ||
| Sponsored Links |
|
#2
21-08-2016, 14:57
|
||||
|
||||
|
This is nice i'll test.
Update............. Nice speed. I was tried chunks like 50m or 1g and 8b and the percent freeze in 9% or 100%. But in 1024k works very well. Maybe only accept k in chunks. Also pack and unpack no errors. Tested with 80 gb folder. Threads 6 level 21 chunks 1024k we need the felice2011 benchmark here!!! Last edited by ChronoCross; 21-08-2016 at 18:24.
|
|
#3
21-08-2016, 22:08
|
||||
|
||||
|
Quote:
 ZStd is also an LZ-Compressor which uses about 10-20 times your Chunk size for better results. So, If ya enter 1 GB as chunk size, you'll need 20 GB RAM
__________________
We are such stuff as dreams are made on, and our little life is rounded with a sleep.
|
|
#4
22-08-2016, 06:03
|
||||
|
||||
|
Ok thanks for the tip.
With 8b the progress stack in 9% and complete the job without errors. In 50m percent goes quick to 100% and stack to complete the job. In 1g arc.exe crash. In any case works this cls works faster.
|
|
#5
22-08-2016, 09:01
|
||||
|
||||
|
1G chunks? Lol
And using 6 threads. That's 1GBx2x6 = 12GB Ram usage. X86 is limited to 2GB. Memory usage is calculated as chunk size x 2 x number of threads
|
| The Following 3 Users Say Thank You to Razor12911 For This Useful Post: | ||
|
#6
22-08-2016, 09:17
|
||||
|
||||
|
Hey wow ok. it's a test men. My tests goes from 0 to the limit of the program.
|
|
#7
22-08-2016, 13:48
|
||||
|
||||
|
I place an quick test of compression and decompression with enwik8, with and without cls.dll, with a block of memory resident in the 16384k heap, to equal the same compression ratio in response to the use of zstd without the cls.
I also tried to use 8 threads, the speed of compression increases considerably, but the cpu ... squirts to hell, with freeze pc if not adequately cooled. zstd without cls Code:
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:18:24
==================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○
Arc 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:42:60 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\
-mzstd
==================================================================================================================================
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:20:16
==================================================================================================================================
○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○
Arc ...\zstd_exe.arc 26,2°MB 00:00:00:36 x -o+ -w.\
==================================================================================================================================
Code:
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 22/08/2016 19:28:24
==================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○
Arc 95,4°MB 26,1°MB 27,36 % 1°Files 00:00:18:57 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -w.\
-mzstd
==================================================================================================================================
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Decompressed Archive Completed At -↓- 22/08/2016 19:31:16
==================================================================================================================================
○ [ ALGORITHM ] ○ [ ARCHIVE NAME ] ○ [ INPUT SIZE ] ○ [ DECOMP.TIME ] ○ [ ARGUMENTS - OPTIONS ] ○
Arc ...\zstd_cls.arc 26,1°MB 00:00:00:43 x -o+ -w.\
==================================================================================================================================
Code:
[CLS] Level=22 ChunkSize=16384k Threads=4 On the contrary, the decompression time has a lower speed of 15-17% with the use of cls, but in this case I think you should test larger files. These dll alternative by razor are a real boon from sky ...
__________________
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « I Mediocri Imitano, I Geni Copiano, Dio Crea & Distrugge » (Io Ridefinisco & Perfeziono le Loro Opere Rendendole Uniche)   ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « Mediocrities Imitate, Genius Copy, God Creates & Destroys » (I Reconsider & Improve Their Works, Rending Them One And Only)
Last edited by felice2011; 22-08-2016 at 13:53.
|
|
#8
07-10-2016, 07:25
|
||||
|
||||
|
Hello mates, I was not here from long time due to my studies actually, I managed some time today, so thought for a modified version by me in ZSTD
What I did? 1. Actually increased speed around atleast 2mb/s with the same cls through modified zstdlib_x86.dll without any ratio loss. 2. Added latest compiled version of Zstd x32 and x64, additionally PZstd also! 3. I did some benchmarks also, for all tests level 19 was used, was tested for compression only and threads were set as 2(It doesn't matter in PZstd case), I have maximum 2 threads only. Test with new and older DLL Code:
By using zstdlib_x86 Modified For Pzstd ---->>
Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 103.66 sec/real 59.58 sec = 174%. Speed 3.82 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 19:35:17.12
Install Finished - 19:36:17.06
By using zstdlib_x86 Razor For Pzstd ---->>
Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%
Compression time: cpu 163.23 sec/real 98.28 sec = 166%. Speed 2.32 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 19:43:16.40
Install Finished - 19:44:54.83
--------------------------------------------------------------------------------------------------------
By using zstdlib_x86 Razor For Zstd x32 ---->>
Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%
Compression time: cpu 160.95 sec/real 104.04 sec = 155%. Speed 2.19 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 19:23:26.29
Install Finished - 19:25:10.48
By using zstdlib_x86 Razor For Zstd x64 ---->>
Compressed 2,907 files, 227,863,642 => 84,657,403 bytes. Ratio 37.15%
Compression time: cpu 166.00 sec/real 103.52 sec = 160%. Speed 2.20 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 19:26:41.28
Install Finished - 19:28:24.95
-----------------------------------------------------------------------------------------------------------------------
By using zstdlib_x86 Modified For Zstd x32 ---->>
Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 109.63 sec/real 74.61 sec = 147%. Speed 3.05 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 19:30:54.96
Install Finished - 19:32:09.99
By using zstdlib_x86 Modified For Zstd x64 ---->>
Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 106.92 sec/real 76.79 sec = 139%. Speed 2.97 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 19:33:06.21
Install Finished - 19:34:23.35
Tests Between Zstd and Pzstd with the Modified DLL Code:
Pzstd
By using Pzstd ---->>
Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 107.86 sec/real 65.95 sec = 164%. Speed 3.45 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 15:10:28.93
Install Finished - 15:11:35.30
By using Zstd x32 ---->>
Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 105.44 sec/real 66.34 sec = 159%. Speed 3.43 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 15:21:16.69
Install Finished - 15:22:24.00
By using Zstd x64 ---->>
Compressed 2,907 files, 227,863,642 => 84,648,750 bytes. Ratio 37.15%
Compression time: cpu 106.08 sec/real 67.47 sec = 157%. Speed 3.38 mB/s
All OK
*******************************************************************************
Compressing is Completed
*******************************************************************************
Install Started - 15:23:24.16
Install Finished - 15:24:32.64
Edit: I will prefer to use zstd32_gcc620.exe or zstd64_gcc620.exe as zstd.exe for now with current Razor CLS.
__________________
"The possession of knowledge does not kill the sense of wonder and mystery. There is always more mystery." Last edited by ~MAK~; 10-10-2016 at 11:28.
|
| The Following 3 Users Say Thank You to ~MAK~ For This Useful Post: | ||
|
#9
07-10-2016, 07:31
|
||||
|
||||
|
Quote:
|
|
#10
07-10-2016, 09:14
|
||||
|
||||
|
Quote:
__________________
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « I Mediocri Imitano, I Geni Copiano, Dio Crea & Distrugge » (Io Ridefinisco & Perfeziono le Loro Opere Rendendole Uniche) ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « Mediocrities Imitate, Genius Copy, God Creates & Destroys » (I Reconsider & Improve Their Works, Rending Them One And Only)
|
|
#11
07-10-2016, 09:29
|
||||
|
||||
|
Yup. I doubt that there will much diff though.
|
|
#12
08-10-2016, 05:18
|
||||
|
||||
|
Code:
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:54:37
=============================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○
Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:44:940 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
=============================================================================================================================================
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:56:59
=============================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○
Arc+zstd32_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:959 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
=============================================================================================================================================
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 13:59:05
=============================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○
Arc+zstd64_gcc620 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:956 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
=============================================================================================================================================
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 14:09:55
=============================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○
Arc+pzstd_gcc620 95,4°MB 26,7°MB 27,99 % 1°Files 00:00:09:946 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mpzstd
=============================================================================================================================================
Code:
Level=19 ChunkSize=100M Threads=2 Code:
pzstd -19 -f
__________________
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « I Mediocri Imitano, I Geni Copiano, Dio Crea & Distrugge » (Io Ridefinisco & Perfeziono le Loro Opere Rendendole Uniche) ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « Mediocrities Imitate, Genius Copy, God Creates & Destroys » (I Reconsider & Improve Their Works, Rending Them One And Only)
|
|
#13
08-10-2016, 06:57
|
||||
|
||||
|
Quote:
__________________
"The possession of knowledge does not kill the sense of wonder and mystery. There is always more mystery."
|
|
#14
08-10-2016, 08:25
|
||||
|
||||
|
Code:
-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 08/10/2016 17:14:41
=============================================================================================================================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○
Arc+zstd 95,4°MB 26,2°MB 27,46 % 1°Files 00:00:43:973 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mzstd
=============================================================================================================================================
same all settings "-19" performed with enwik8. Try you also with the same input file 
__________________
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « I Mediocri Imitano, I Geni Copiano, Dio Crea & Distrugge » (Io Ridefinisco & Perfeziono le Loro Opere Rendendole Uniche) ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ ≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈ « Mediocrities Imitate, Genius Copy, God Creates & Destroys » (I Reconsider & Improve Their Works, Rending Them One And Only)
|
|
#15
08-10-2016, 08:32
|
||||
|
||||
|
Quote:
__________________
"The possession of knowledge does not kill the sense of wonder and mystery. There is always more mystery." Last edited by ~MAK~; 08-10-2016 at 10:08.
|
 |
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Masked Compression (Ultimate compression in one go) | panker1992 | Conversion Tutorials | 661 | 13-05-2024 16:51 |
| Simple splitting and compress by DLG | felice2011 | Conversion Tutorials | 106 | 14-01-2018 15:32 |
| Reflate+Static+Fazip | 1234567890123 | Conversion Tutorials | 20 | 14-04-2016 20:53 |
| lzma2 on freearc as external | panker1992 | Conversion Tutorials | 39 | 19-12-2015 09:01 |
Linear Mode
