FileForums - Z Standard Best Compression Options

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - Z Standard Best Compression Options (https://fileforums.com/showthread.php?t=102734)

Z Standard Best Compression Options

I want to know the best option to get better Ratio in compression...with Z standard!

I currently use
ZSTD.exe -22 - - ultra -T16

i need to know,
is there any other option to get better Ratio
Like lolz mc128 - mc1023

Please, I need Help form Who using Z standard...

This info For New Guys!!!

Why i Use Z Standard? Cause
Z standard Compress,
much faster then lolz,
that's why i use it.

For example:
lolz takes 8h to compress 50GB to 18GB.
Z standard takes 1h 50m to compress 50GB to 18.5GB

With this option it is even worse then the standard lzma/4x4 option and tooks longer in my tests.

zstd:19:T0 That's enough!

I suggest from

Code:

4x4:lzma:a1:mfbt4:32m:fb273:mc10000:lc8

Code:

lolz:d32:tt1:mtt1:mt8:mc2

Here is a old test set from me for comparison.

Code:

lolz:dtb1:mtt1:mt6:d128m:mc1023

Compressed 1 file, 512,039,065 => 174,850,394 bytes. Ratio 34.15%

Compression time: cpu 0.73 sec/real 278.94 sec = 0%. Speed 1.84 mB/s



lzma:a1:mfbt4:d256m:fb273:mc10000:lc8

Compressed 1 file, 512,039,065 => 235,985,799 bytes. Ratio 46.09%

Compression time: cpu 0.81 sec/real 96.79 sec = 1%. Speed 5.29 mB/s



4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc10000

Compressed 1 file, 512,039,065 => 237,540,041 bytes. Ratio 46.39%

Compression time: cpu 0.69 sec/real 38.33 sec = 2%. Speed 13.36 mB/s



razor:128m

Compressed 1 file, 512,039,065 => 206,659,001 bytes. Ratio 40.36%

Compression time: cpu 1.44 sec/real 528.19 sec = 0%. Speed 0.97 mB/s



zstd -22 --ultra -T2

Compressed 1 file, 512,039,065 => 255,419,289 bytes. Ratio 49.88%

Compression time: cpu 0.70 sec/real 200.09 sec = 0%. Speed 2.56 mB/s

Code:

4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc10000

Compressed 1 file, 512,039,065 => 237,552,373 bytes. Ratio 46.39%

Compression time: cpu 0.58 sec/real 32.19 sec = 2%. Speed 15.91 mB/s



4x4:lzma:a1:mfbt4:32m:fb273:mc10000:lc8

Compressed 1 file, 512,039,065 => 240,037,512 bytes. Ratio 46.88%

Compression time: cpu 0.58 sec/real 25.33 sec = 2%. Speed 20.21 mB/s

Code:

lolz:d32:tt1:mtt1:mt8:mc2 (pretty fast and only ~2% less ratio then my standard settings. I think this is a good choice. What about decompression speed?)

Compressed 1 file, 512,039,065 => 185,075,365 bytes. Ratio 36.14%

Compression time: cpu 0.56 sec/real 71.90 sec = 1%. Speed 7.12 mB/s



lolz:dtb1:mtt1:mt6:d128m:mc1023

Compressed 1 file, 512,039,065 => 174,726,950 bytes. Ratio 34.12%

Compression time: cpu 0.70 sec/real 279.31 sec = 0%. Speed 1.83 mB/s

Code:

Same as "lolz:d32:tt1:mtt1:mt8:mc2" but with d128

Compressed 1 file, 512,039,065 => 183,759,891 bytes. Ratio 35.89%

Compression time: cpu 0.45 sec/real 92.19 sec = 0%. Speed 5.55 mB/s

About the lzma and lolz i know,
thanks to Simorq and KaktoR :) and doofoo24

Z Standard is New to me,
and ZSTD Comp & Decomp fast,
that's why i like it...

i like to know more about it.

Code:

*** zstd command line interface 64-bits v1.3.8, by Yann Collet ***

Usage :

      ZSTD [args] [FILE(s)] [-o file]



FILE    : a filename

          with no FILE, or when FILE is - , read standard input

Arguments :

 -#     : # compression level (1-19, default: 3)

 -d     : decompression

 -D file: use `file` as Dictionary

 -o file: result stored into `file` (only if 1 input file)

 -f     : overwrite output without prompting and (de)compress links

--rm    : remove source file(s) after successful de/compression

 -k     : preserve source file(s) (default)

 -h/-H  : display help/long help and exit



Advanced arguments :

 -V     : display Version number and exit

 -v     : verbose mode; specify multiple times to increase verbosity

 -q     : suppress warnings; specify twice to suppress errors too

 -c     : force write to standard output, even if it is the console

 -l     : print information about zstd compressed files

--ultra : enable levels beyond 19, up to 22 (requires more memory)

--long[=#]: enable long distance matching with given window log (default: 27)

--fast[=#]: switch to ultra fast compression level (default: 1)

--adapt : dynamically adapt compression level to I/O conditions

 -T#    : spawns # compression threads (default: 1, 0==# cores)

 -B#    : select size of each job (default: 0==automatic)

 --rsyncable : compress using a rsync-friendly method (-B sets block size)

--no-dictID : don't write dictID into header (dictionary compression)

--[no-]check : integrity check (default: enabled)

 -r     : operate recursively on directories

--format=zstd : compress files to the .zst format (default)

--format=gzip : compress files to the .gz format

--test  : test compressed file integrity

--[no-]sparse : sparse mode (default: enabled on file, disabled on stdout)

 -M#    : Set a memory usage limit for decompression

--no-progress : do not display the progress bar

--      : All arguments after "--" are treated as files



Dictionary builder :

--train ## : create a dictionary from a training set of files

--train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args

--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fast cover algorithm with optional args

--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)

 -o file : `file` is dictionary name (default: dictionary)

--maxdict=# : limit dictionary to specified size (default: 112640)

--dictID=# : force dictionary ID to specified value (default: random)



Benchmark arguments :

 -b#    : benchmark file(s), using # compression level (default: 3)

 -e#    : test all compression levels from -bX to # (default: 1)

 -i#    : minimum evaluation time in seconds (default: 3s)

 -B#    : cut file into independent blocks of size # (default: no block)

--priority=rt : set process priority to real-time

zstd --long is alternative to srep like ldmf in lolz...
i tested --ultra -22 --zstd=wlog=31,clog=30,hlog=30 worst than srep+zstd --ultra -22...
just use zstd --ultra -22 with normal srep:m3f:d1g:a2...

Thanks Simorq, KaktoR, doofoo24

guys i need Z standard Decompressor
x86 & x64 ZSTD version 1.3.7.0 Decompressor..

Please upload it..

Quote:

Originally Posted by yasitha (Post 480667)

Thanks Simorq, KaktoR, doofoo24

guys i need Z standard Decompressor
x86 & x64 ZSTD version 1.3.7.0 Decompressor..

Please upload it..

Why do you still want to use zstd, when people already posted results with SMALLER size and FASTER decompression with 4x4:lzma?

Quote:

Originally Posted by FitGirl (Post 480669)

Why do you still want to use zstd, when people already posted results with SMALLER size and FASTER decompression with 4x4:lzma?

Problem is why people still want to use zstd when games themselves are packed using oodle ??

I use 90% lolz 10% zstd. ��

Code:

-22 --ultra -T1

best compress method for Zstandard i guess :D

Quote:

Originally Posted by st0rm (Post 480678)

Code:

-22 --ultra -T1

best compress method for Zstandard i guess :D

I need decompress exe files

ZSTD.exe (v1.3.7.0) 32bit and 64 bit for decomp

Can u upload it?

Zstandard v1.4.0

Code:

[External compressor:zstandard]

header    = 0

default   = -22 --ultra -T1

packcmd   = zstd.exe {options} <stdin> <stdout>

unpackcmd = zstd.exe -d -f <stdin> <stdout>

try this for pack and unpack.

Quote:

Originally Posted by Simorq (Post 480680)

Code:

https://github.com/facebook/zstd/rel....4.0-win32.zip

https://github.com/facebook/zstd/rel....4.0-win64.zip

Thanks Simorq :)

Today i use it for Hitman Results is Excellent...

i did a test with LZMA & ZSTD result is :

Used Method : afr+srep:m3f+zstd:ultra:22:T1

Code:

FreeArc 0.67 (March 15 2014) extracting archive: forge_zstd.arc

Extracted 18 files, 2,012,040,342 => 7,460,454,400 bytes. Ratio 26.97%

Extraction time: cpu 735.30 sec/real 253.38 sec = 290%. Speed 29.44 mB/s

All OK

Used Method : afr+srep:m3f+lzma:ultra

Code:

FreeArc 0.67 (March 15 2014) extracting archive: forge_lzma.arc

Extracted 18 files, 1,901,738,484 => 7,460,454,400 bytes. Ratio 25.49%

Extraction time: cpu 913.78 sec/real 309.55 sec = 295%. Speed 24.10 mB/s

All OK

Game name : Prine of Persia [2008]

ya it's expected with precompressor, zstd ratio worst by a lot compare to 4x4:lzma...
where i found zstd great is when used only with srep like srep+zstd...
for example games like hitman/ the evil within/Wolfenstein The Old Blood...

Quote:

Originally Posted by doofoo24 (Post 480742)

Example from Hitman 2

Code:

srep+zstd:22:ultra:T2

Compressed 1 file, 2,274,680,007 => 1,363,512,459 bytes. Ratio 59.94%

Compression time: cpu 3.08 sec/real 461.75 sec = 1%. Speed 4.93 mB/s

All OK



srep+4x4:LZMA

Compressed 1 file, 2,274,680,007 => 1,279,557,434 bytes. Ratio 56.25%

Compression time: cpu 4.80 sec/real 175.97 sec = 3%. Speed 12.93 mB/s

All OK



srep+LZMA

Compressed 1 file, 2,274,680,007 => 1,270,735,418 bytes. Ratio 55.86%

Compression time: cpu 5.33 sec/real 462.91 sec = 1%. Speed 4.91 mB/s

All OK



srep+lolz:dtb1:mtt1:mt6:d128m:mc1023

Compressed 1 file, 2,274,680,007 => 1,199,850,492 bytes. Ratio 52.75%

Compression time: cpu 2.47 sec/real 1071.78 sec = 0%. Speed 2.12 mB/s

All OK

Quote:

Originally Posted by KaktoR (Post 480747)

Example from Hitman 2

Code:

srep+zstd:22:ultra:T2

Compressed 1 file, 2,274,680,007 => 1,363,512,459 bytes. Ratio 59.94%

Compression time: cpu 3.08 sec/real 461.75 sec = 1%. Speed 4.93 mB/s

All OK



srep+4x4:LZMA

Compressed 1 file, 2,274,680,007 => 1,279,557,434 bytes. Ratio 56.25%

Compression time: cpu 4.80 sec/real 175.97 sec = 3%. Speed 12.93 mB/s

All OK



srep+LZMA

Compressed 1 file, 2,274,680,007 => 1,270,735,418 bytes. Ratio 55.86%

Compression time: cpu 5.33 sec/real 462.91 sec = 1%. Speed 4.91 mB/s

All OK



srep+lolz:dtb1:mtt1:mt6:d128m:mc1023

Compressed 1 file, 2,274,680,007 => 1,199,850,492 bytes. Ratio 52.75%

Compression time: cpu 2.47 sec/real 1071.78 sec = 0%. Speed 2.12 mB/s

All OK

i recommend you to do some testing more, also introduce lolz with ldmf,

[External compressor:lolz]
default = -d304m -mt2
header = 0
packcmd = "lz\lolz_x64.exe" {options} -al1 -mc1023 -lm0 -rt0 -dt1 -mtb64m -dtb0 -x0 -tt8 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = "lz\lolz_x64.exe" $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

[External compressor:lolzi]
default = -d512m -mt1
header = 0
packcmd = "lz\lolz2_x64.exe" {options} -ldmf1 -ldc0 -al1 -mc1023 -lm0 -rt0 -dt1 -mtb64m -dtb0 -x0 -tt12 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = "lz\lolz2_x64.exe" $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

and i also recommend to check these settings as well :D

it's not wise to have mmt1 enabled, and its not wise to have many threads on lz type of algorithms as well,

lastly when doing tests compare apples to apples

ZSTD (--long=31) optional --ultra -22 mode only support 1 Thread even if you use more !!!!!