SuperREP (SREP) Full Info!
 

First of all 😊
🎉Merry 🎅Christmas🎉

My Question is⚠️
I Like to know about Srep Commands and methods, 😉
i already try Google, and Fileforums, and some others, and i have found this,
and its not much.☹️ so that's why i ask you guys to help me out :)

http://krinkels.org/resources/superrep-srep.107

https://www.softpedia.com/get/Compre...SuperREP.shtml

I have no idea about Srep,
that's why i ask, and it's really important to me,
Otherwise I don't ask. ✔️

In my experience, i have Try Srep with Lzma,
for testing which one do better.

Those are the method
that i have tried..

srep:m3f:l128:a1+LZMA
srep:m5f:l128:a1+LZMA

srep:m3f:l512:a1+LZMA
srep:m5f:l512:a1+LZMA

and I see l512 is faster then l128, but i didn't see m5/m3 different, and i know *a1* can go up to *a64* but before i do that i like to know about all srep commands and what they do by increasing and decreasing.

So Please Guys Help me out :)

Run "srep64" from the command line is all written in the info.
Merry Christmas to you too.

this is what I use it is for me is the best. if you want to understand how it works run getparam batch file. Merry Christmas

What do you mean by Command lin?
you mean srep64.exe open with CMD?

Thanks for Reply 🤗

Thanks I'll try that soon.. & reply back 🎅

Yes, write only srep64 and run cmd

Thanks.
Will report back thanks again.. 🤗

I have simple batch files for all of the tools I use

info.bat

Nice, can i have that?
All samples? 🎄

Thanks a lot. 🤗

can you help me with something?

I know how to find Oodle games,
Oodle games have dll file inside the game and Oodle scan tells us which method that game used,
Kraken, Mermaid, Selkie. Jc4 used K&M ACO use Mermaid only. Please Correct me if I'm wrong 🤗

also i need to know, On none Oodle games
How can i find what kind of Compression,
used for the game?

It's very important to me to make best compression method for every game 🎮

What do you mean by default settings?
Please explain..

Thanks for Reply🙂

Yes your Right, i have use m5f:l512:a16 and it took 6 Hours for Srep to complete.

So your saying if i use
srep:m5f:l128:1a VS srep:m3f
Between them it's just a 2% different like 500Mb lower then other..?

can you,
Recommend Best SREP Settings for all games.

What that a2 does?
i have seen the info about Srep,
it's only says
*a0 is slowest but requires least memory..
Please explain.

I7 8700K.
Corsair Vengeance 3200MHz 8GB x 4 (32GB) DDR4.

i can use all 32GB Ram for compressing,
but i want low ram using for decompressing.

I don't want to limit ram usage for decompressing.
i want to know if i use lot of ram for compressing it will require the same amount of ram for decompressing..

so how much it will require for decompressing?
it will auto detect ram amount,
then it use as much as ram for decompressing?

Link please!

so what your recommendation?
what method should i use?
Please provide me the best Srep configuration for
high compression and faster.

Oky thnaks for your help.. I appreciate it.
Have a great day :)

Question Update
 

Now I know about Srep

SREP PROBLEM is SOLVED
NO MORE QUESTIONS

I Like To Thanks All of You!

felice2011
oltjon
KaktoR
Simorq
doofoo24

THANKS TO ALL OF YOU FOR HELP

Now i have LZMA Question.
so i will open LZMA thread for it.!

I have open new Thread can you post there :) please.

i saw most of the tools use Srep v3.92
Even UltraARC, but srep v3.93 is available,

Why they don't use the latest version of srep?

because 3.92 stable version

Oo i see Thanks :)

bit what about this one?
cls-srep-R2
https://fileforums.com/showthread.php?t=97976

This one not stable?

SuperREP: huge-dictionary LZ77 preprocessor
 

Description
SuperREP is the first LZ77 compressor that supports dictionaries larger than RAM available. Default settings (-l512) allows to process files that are 20x larger than RAM size. Memory requirements are proportional to 1/L, so by increasing -l value it's possible to process even larger files. Compression speed is 100 mb/s and decompression runs at 250 mb/s on i3-2100.

User reports
First (23% better compression compared to REP)
Second (31% better compression compared to REP)
Third (21% better compression compared to REP)
Fourth (45% better compression compared to pure LZMA)

LZ77 algorithm
-m1..-m3 compression modes implement LZ77 compression algorithm.

-m1: input file split into chunks of L bytes (specified by -c option, 512 bytes by default). For every chunk, the program stores SHA-1 hash. When later it encounter L-byte chunk with the same SHA-1 value, it replaces new chunk with reference to previous one, assuming that chunks are equal.

-m2: same as -m1, but only weak hash stored for every chunk. When the program encounters chunk with the same hash value, it rereads old chunk from input file in order to compare data.

-m3: same as -m2, but the program compares bytes before and after equal chunks in order to extend match as much as possible. -l option may be used to specify minimum match length.

On decompression, every squeezed chunk is restored by reading the contents of previous equal chunk from output file. The compression algorithm in -m2/m3 modes rereads the same chunks in order to compare them with current data. This puts heavy load on OS I/O system and disk cache.

The algorithm requires that input file when compressing (except for -m1 mode) and output file when decompressing is seekable. If that's not true, you may use -temp option to tell the program to create temporary file used to store a copy of all uncompressed data.

The algorithm also need to know size of input file in advance when compressing. If the program cannot determine the file size (i.e. when compressing from stdin), file size should be supplied via -s option. Values larger than actual filesize will work as well, and by default 25gb is assumed.

Future-LZ algorithm
-m1f..-m3f compression modes implement Future-LZ compression algorithm.

Future-LZ is a modification of LZ77 storing matches at the match source rather than destinatioon position. For compressor like SREP that utilizes only long matches it allows to decrease amount of data stored in the decompressor, therefore decreasing amount of memory required for decompression. In my tests, decompression required RAM equal to about 10% of filesize. Moreover, since we know order of access to these stored data, they may be swapped from RAM to diskfile without losing efficiency, so decompression may be performed using just about 100 mb of RAM.

Future-LZ compression is performed in two passes - at the first pass, matches are found and saved in the memory. Then matches are sorted by their source address and the second pass performed, storing each match in the compressed file at position of match source. So, Future-LZ compression requires seekable input file on compression stage. Decompression access both input and output files sequentially, thus very fast.

Interpreting decompression stats for Future-LZ mode

Let's consider the following line:

Matches 1073 3119 9752, I/Os 0, RAM 346/1024, VM 664/984, R/W 824/1488

First 3 numbers denote current, maximum so far, and total number of matches in dictionary.

I/Os is a number of long matches that were copied directly form decompressed file (-m option).

Other numbers mean that there are 1024 MiB of RAM allocated, of those 346 MiB is currently used (memory is never returned to OS).
VM file is 984 MiB long, of those 664 MiB is in use now.
1488 MiB were written to VM file, of those 824 MiB was already read back.
VM.current = VM.W - VM.R equation is always true, here it's 664=1488-824.

At the end of decompression, we have a sort of that:

RAM 0/983, VM 0/1000, R/W 1560/1560

i.e. two zeros and R=W, while the rest of numbers shows how much memory/disk was used and how much data were written (and read back) to VM file.

Also, sum of RAM and VM size required for given file decompression should be constant. Actually, it slightly grows as -mem decreased, due to inefficiencies in the memory management.

Please also note that -mem option limits total amount of RAM used, that includes 40 mb for I/O buffers, while RAM value in stats shows only memory used for match data.

Memory usage for compression

When compressing, memory usage:
-m1: 20*filesize/L + 4*filesize/L + roundup(5*filesize/L) + roundup(A*filesize/L) + 48 mb
-m2/3: 4*filesize/L + roundup(5*filesize/L) + roundup(A*filesize/L) + 48 mb
where roundup() rounds its argument up to the next power of 2, L is value of -c option, and A is value of -a option. So, overall, memory allocated in 4-5 chunks.

Memory usage for decompression

Decompression of files compressed with LZ77 requires only 24mb for I/O buffers and no hash. Repeated data are copied directly from output file, though, so you need to have enough RAM used for disk cache in order to make decompression fast.

Decompression of files compressed with Future-LZ prints amount of memory required (it depends on many factors). You may limit amount of RAM used for decompression using -mem option:
-mem75% means "use no more than 75% of RAM" - that's the default setting
-mem75p means the same
-mem600mb means itself
-mem75%-600mb means "use no more than 75% of RAM minus 600 mb"
Extraneous data will be placed into the file specified by -vmfile option.
Also you may decrease amount of required RAM by using -m option to copy longer matches directly from output file.

Temporary file

Temporary file created automatically when necessary. You may disable its usage by "-temp=" option w/o parameter.

How to set up FreeArc to use SREP in filter mode

In order to use SREP as external compressor in FreeArc, add one of the following sections to your arc.ini:

This section is optimized for srep:f (Future-LZ) compression, therefore compressed data are sent immediately to stdout:

[External compressor:srep]
packcmd = srep {options} $$arcdatafile$$.tmp - <stdout>
unpackcmd = srep -d {options} - - <stdin> <stdout>

This section is optimized for LZ77 compression:

[External compressor:srep]
packcmd = srep {options} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = srep -d {options} - - <stdin> <stdout>

Some explanations for arc.ini settings:

Compression:

1. SREP should know infile size since it allocates memory proportional to filesize at the beginning of compression. If it doesn't know filesize, 25 gb (-s option) is assumed, so it allocates a lot of memory (1-2 gb).

It's why i don't have used here - if freearc first writes data to the $$arcdatafile$$.tmp, then srep will know its size and allocate just the amount of memory required. If you can afford allocating 1-2 gb of memory (and you never compress with srep more than 25 gb), then you can use mode that will make process a bit faster (20-40 seconds per gb).

2. Except for plain -m1 mode, freearc needs to reread input data again: in -m2/m3 mode it checks matches by rereading input file, and in -f mode it performs 2 passes - first one to find matches, second one to encode them. SREP automatically uses tempfile to store copy of input data when data are read from stdin and they will be required later (i.e. in all modes except for -m1), so it will work anyway but not much faster than $$arcdatafile$$.tmp mode.

3. If srep writes data to stdout, next compression algorithms may be performed in parallel. this means that both srep and lzma will use memory at the same time so total memory usage will be increased. some people don't have enough ram for such usage. it's why my default settings for LZ77 modes don't use stdin/stdout.

4. With Future-LZ, we perform 2 passes: 1) we read input data and find matches using lot of memory, 2) we read data again and produce compressed file using minimum of memory. so we can perform second pass simultaneously with lzma and it's why i recommend to use mode with Future-LZ.

Decompression:

1. If data was compressed without -f, srep will copy matches from output file. if output data are written to stdout, duplicate tempfile will be created.

2. With -f, srep allocates amount of memory specified by -mem option, leabing less memory for lzma decompression.

In both cases, decompression from stdin to stdout usually will be better than using files.

new question...

If i compress big games like 80GB
Do i have to chage this settings?

sBYTES: explicitly specify filesize (for compression from stdin), default 25gb

-s helps srep preallocate size for dictionary and other stuff, it's nice idea to specify the correct sizeif you're using srep at first in the compression chain and at least twize the size if you're using any recompressor like xtool before it

-s value directly affects compression memory and specifying less value will mae srep fail during compression which can be really frustrating when compressing large files

Thanks..