Simple splitting and compress by DLG

[panker1992]

Originally Posted by RamiroCruzo

Amigos if you wanna so fast compression method, why don't you take a look at LZ4 which goes beyond 1GB/s without I/O overwrapping & multithreading.....

I have tested lz4 and lz5 long time ago, zstd is a step ahead of lz4 lz5 and zhuff at the moment its steady and it has a lot of support the one i posted is v3.6 and the one i am testing now is 4.2 and it has 2.2 gigs/sec around
( 593mb/sec/thread = 2.2 gigs/sec )

[felice2011]

Originally Posted by RamiroCruzo

Amigos if you wanna so fast compression method, why don't you take a look at LZ4 which goes beyond 1GB/s without I/O overwrapping & multithreading.....

Originally Posted by panker1992

I have tested lz4 and lz5 long time ago, zstd is a step ahead of lz4 lz5 and zhuff at the moment its steady and it has a lot of support the one i posted is v3.6 and the one i am testing now is 4.2 and it has 2.2 gigs/sec around
( 593mb/sec/thread = 2.2 gigs/sec )

Friends you do your choices, I'm happy to update my utility just as you wish, and it however you like.
Panker do not look at only the speed, but also the compression level of the various methods, we must try to find a balance in both, to satisfy everyone.

[panker1992]

Originally Posted by felice2011

Friends you do your choices, I'm happy to update my utility just as you wish, and it however you like.
Panker do not look at only the speed, but also the compression level of the various methods, we must try to find a balance in both, to satisfy everyone.

This is why i chose zstd ...because it features advanced API that can compress decent.

To be more specific zstd has at least 30% ~ 35% better compression ratio
tho everyone i see use my original -8 switch ....it can easily go as high as -20 but its ratio doesnt compare to lzma at all :P

All in all zstd gives 30% ~ 35% better compression AND it decompressed as fast as lz4

HINT : zstd and all tools i use have porpose nothing i choose is coincidence this all is part of my upcoming big project call "Masked Compression"

soon !!!

[RamiroCruzo]

Originally Posted by felice2011

Friends you do your choices, I'm happy to update my utility just as you wish, and it however you like.
Panker do not look at only the speed, but also the compression level of the various methods, we must try to find a balance in both, to satisfy everyone.

Lets compare ourselves which is better for our needs....

Source:Encode.ru

| Compressor name | Compression| Decompress.| Compr. size | Ratio |
| --------------- | -----------| -----------| ----------- | ----- |
| memcpy | 8533 MB/s | 8533 MB/s | 104857600 |100.00 |
| lz4 r131 | 480 MB/s | 2275 MB/s | 64872315 | 61.87 |
| lz4hc r131 -1 | 82 MB/s | 1896 MB/s | 59448496 | 56.69 |
| lz4hc r131 -3 | 54 MB/s | 1932 MB/s | 56343753 | 53.73 |
| lz4hc r131 -5 | 41 MB/s | 1969 MB/s | 55271312 | 52.71 |
| lz4hc r131 -7 | 31 MB/s | 1969 MB/s | 54889301 | 52.35 |
| lz4hc r131 -9 | 24 MB/s | 1969 MB/s | 54773517 | 52.24 |
| lz4hc r131 -11 | 20 MB/s | 1969 MB/s | 54751363 | 52.21 |
| lz4hc r131 -13 | 17 MB/s | 1969 MB/s | 54744790 | 52.21 |
| lz4hc r131 -15 | 14 MB/s | 2007 MB/s | 54741827 | 52.21 |
| lz5 r131 | 195 MB/s | 939 MB/s | 55884927 | 53.30 |
| lz5hc r131 -1 | 32 MB/s | 742 MB/s | 52927122 | 50.48 |
| lz5hc r131 -3 | 20 MB/s | 716 MB/s | 50970192 | 48.61 |
| lz5hc r131 -5 | 10 MB/s | 701 MB/s | 49970285 | 47.66 |
| lz5hc r131 -7 | 5.54 MB/s | 682 MB/s | 49541511 | 47.25 |
| lz5hc r131 -9 | 2.69 MB/s | 673 MB/s | 49346894 | 47.06 |
| lz5hc r131 -11 | 1.36 MB/s | 664 MB/s | 49266526 | 46.98 |
| zstd v0.3 | 257 MB/s | 547 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -1 | 257 MB/s | 553 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -3 | 76 MB/s | 417 MB/s | 46774383 | 44.61 |
| zstd_HC v0.3 -5 | 40 MB/s | 476 MB/s | 45628362 | 43.51 |
| zstd_HC v0.3 -9 | 14 MB/s | 485 MB/s | 44840562 | 42.76 |
| zstd_HC v0.3 -13 | 9.34 MB/s | 469 MB/s | 43114895 | 41.12 |
| zstd_HC v0.3 -17 | 6.02 MB/s | 463 MB/s | 42989971 | 41.00 |
| zstd_HC v0.3 -21 | 3.35 MB/s | 461 MB/s | 42956964 | 40.97 |
| zstd_HC v0.3 -23 | 2.33 MB/s | 463 MB/s | 42934217 | 40.95 |

[ShadowEagle]

Used this method:



I tried to extract with inno setup. I copied all files from "Resources" folder into includes folder in inno setup script and added the reflate files into the script:

Source: Include\reflate.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\rawrest.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\rawdet.exe; DestDir: {tmp}; Flags: dontcopy
Source: Include\raw2hif.exe; DestDir: {tmp}; Flags: dontcopy

But i get this error:



Pls help. What i`m doing wrong?

[panker1992]

Originally Posted by RamiroCruzo

Lets compare ourselves which is better for our needs....

Source:Encode.ru

| Compressor name | Compression| Decompress.| Compr. size | Ratio |
| --------------- | -----------| -----------| ----------- | ----- |
| memcpy | 8533 MB/s | 8533 MB/s | 104857600 |100.00 |
| lz4 r131 | 480 MB/s | 2275 MB/s | 64872315 | 61.87 |
| lz4hc r131 -1 | 82 MB/s | 1896 MB/s | 59448496 | 56.69 |
| lz4hc r131 -3 | 54 MB/s | 1932 MB/s | 56343753 | 53.73 |
| lz4hc r131 -5 | 41 MB/s | 1969 MB/s | 55271312 | 52.71 |
| lz4hc r131 -7 | 31 MB/s | 1969 MB/s | 54889301 | 52.35 |
| lz4hc r131 -9 | 24 MB/s | 1969 MB/s | 54773517 | 52.24 |
| lz4hc r131 -11 | 20 MB/s | 1969 MB/s | 54751363 | 52.21 |
| lz4hc r131 -13 | 17 MB/s | 1969 MB/s | 54744790 | 52.21 |
| lz4hc r131 -15 | 14 MB/s | 2007 MB/s | 54741827 | 52.21 |
| lz5 r131 | 195 MB/s | 939 MB/s | 55884927 | 53.30 |
| lz5hc r131 -1 | 32 MB/s | 742 MB/s | 52927122 | 50.48 |
| lz5hc r131 -3 | 20 MB/s | 716 MB/s | 50970192 | 48.61 |
| lz5hc r131 -5 | 10 MB/s | 701 MB/s | 49970285 | 47.66 |
| lz5hc r131 -7 | 5.54 MB/s | 682 MB/s | 49541511 | 47.25 |
| lz5hc r131 -9 | 2.69 MB/s | 673 MB/s | 49346894 | 47.06 |
| lz5hc r131 -11 | 1.36 MB/s | 664 MB/s | 49266526 | 46.98 |
| zstd v0.3 | 257 MB/s | 547 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -1 | 257 MB/s | 553 MB/s | 51231016 | 48.86 |
| zstd_HC v0.3 -3 | 76 MB/s | 417 MB/s | 46774383 | 44.61 |
| zstd_HC v0.3 -5 | 40 MB/s | 476 MB/s | 45628362 | 43.51 |
| zstd_HC v0.3 -9 | 14 MB/s | 485 MB/s | 44840562 | 42.76 |
| zstd_HC v0.3 -13 | 9.34 MB/s | 469 MB/s | 43114895 | 41.12 |
| zstd_HC v0.3 -17 | 6.02 MB/s | 463 MB/s | 42989971 | 41.00 |
| zstd_HC v0.3 -21 | 3.35 MB/s | 461 MB/s | 42956964 | 40.97 |
| zstd_HC v0.3 -23 | 2.33 MB/s | 463 MB/s | 42934217 | 40.95 |

Impressive but this test utilize the HC format of the zstd program
currently i test zstd v4.2

i have eaten all my day to test, test and test this :P

download the lz4 installer for windows and rename zstd ~~> lz4.exe and
replace in programs/lz4 folder

use this and you will see that decomp speed is all but fake i/o cant handle
2.2 gigs/sec because of limitation unless you have some x6 SSD array :P