Shelwien's Reflate compatible with Freearc

[Razor12911]

Files needed for compression:
cls-reflate.dll
reflate.exe

Files needed for decompression:
cls-reflate.dll
reflate.exe

Drawbacks:
It's actually slower during compression process than normal.
-Haven't checked decompression but it's easy to get 6mb/s.

Advantages:
Shows progress when compressing and decompressing.

[Mini]

where control of level?!

[Simorq]

Work Win7 64Bit

[~MAK~]

where control of level?!

Ahh matey, its not even his reflate or he don't even have the source he just tried to make hif and unp files in one file so that freearc could decompress it, because it just support one file input and one file output, I hope you understand, still you should thank him for his try for you all ;)

[gozarck]

Progress works fine, its stuck at about 13 MB each time.

Specifically, it stucks when Reflate has I/O Read = Input size. I think that's a problem cause Stdin and Stdout are based upon Read from input stream handle.

Anyways, nice work bro

[Razor12911]

The problem is from cls, something has to be done for it to work, I know how to fix it but I really don't have time at the moment, my exams are about to start.

[ChronoCross]

Ahh matey, its not even his reflate or he don't even have the source he just tried to make hif and unp files in one file so that freearc could decompress it, because it just support one file input and one file output, I hope you understand, still you should thank him for his try for you all ;)

yeah! That's right! Good try Razor.

[Razor12911]

Release 1 uploaded

CLS Fixed

There may be issues
Only ran test on one input

[Razor12911]

Level 9 is strong compression mate, take zstd, set level 1, how fast is it? set level 22? how fa.. how slow is it?

[Gupta]

Level 9 is strong compression mate, take zstd, set level 1, how fast is it? set level 22? how fa.. how slow is it?

okay i understood what u r saying, bt it is very fast on Characters.vpp_pc, Rockstar and Deep silver both uses same level i.e. 9

[Razor12911]

dude, Characters.vpp_pc, just look at the ratio, it can only be a small sector that had deflate streams within the file, let's say the first 20mb only had deflate streams, the rest of the file has nothing to process so it's just copied as it is without any further restoration because it wasn't processed in the first place because it didn't have deflate streams.
When you look at GTAV, firstly, it has maximum compressed streams, it has streams all over the file, just look at inflated ratios as well.

[Simorq]

x64w.rpf (893MB) x64w.ref (1.99GB)

FreeArc 0.67 (March 15 2014) Creating Archive: Data.arc
Compressed 1 file, 937,340,928 => 2,140,205,962 bytes. Ratio 228.33%
Compression time: cpu 2.04 sec/real 447.45 sec = 0%. Speed 2.09 mB/s
All OK

FreeArc 0.67 (March 15 2014) Extracting Archive: Data.arc
Extracted 1 file, 2,140,205,962 => 937,340,928 bytes. Ratio 228.33%
Extraction time: cpu 5.51 sec/real 164.90 sec = 3%. Speed 5.68 mB/s
All OK

[Mini]

control of level - very good, thanks! there are plans to add multithreading?

[ChronoCross]

Good good! I have the same size like reflateMT.

[Razor12911]

I don't plan to add MT feature, it's a lot of work.

[felice2011]

OK to the disadvantages, but unlike reflateMT already tested, which advantages? only shows progress when compressing and decompressing?

[Razor12911]

Well I really don't know, only thing that I did was make it show progress when using freearc, since there were 3 descriptors instead of 2, I just combined them somehow, and that's it.
Plus it gets updates almost everywhere or whenever there is a bug that was detected.
http://nishi.dreamhosters.com/u/rawfilt_v1l.rar
Source: http://encode.ru/threads/1399-reflate-a-new-universal-deflate-recompressor/page4
This came today, just replace reflate.exe with new one, it will probably work with old one.

[felice2011]

OK thanks

[Blade2239]

Can anybody tell me which relate is perfect to use...
Fast
Great compression
And most reliable...

[vinay sawan]

how it works with inno setup decompression

[Razor12911]

CLS updated

Changes

-Updated for v1l
-Source included

[Razor12911]

Can anybody tell me which relate is perfect to use...
Fast
Great compression
And most reliable...
This is the one to use, very reliable, not sure about speed though. Great compression... nah, what you're looking for is great precompression.

how it works with inno setup decompression

simple straight forward, just add cls and exe in inno, done.

[zouzouni]

First of all I shall thk you Razor for this awesome work and amazing tools you share with us!

Testing report rawfilt v1l:

I could recompress 2x1.8Gb game files using reflate (v1l)+srep+exe2+delta+lzma... that was pretty fast (30 min)

I could also decompress it ok in asc and it is strangely very fast ...3.6 Gb install in less than 2 min ....!!!! burst speed 200Mb/s and average 30Mb/s! ...very impressive!

backsides/limits:

I could compress one 2.5Gb game file ok but decompression stops at 40% helper for setup still runs for a while and stops as well...no error but does nothing, just stops install

is there any size limitation "per file" being processed? it seems after some testing, that it can compress files bigger than 2Gb but it doesn't decompress them...

thus, I tried to recompress 6 Gb (4x1.5Gb files) and comp went ok but once again decomp/install stops randomly at either 14/15% ...

Any repack test ouptut under 4 Gb, with files smaller than 2Gb installs ok with asc setup,
other than that it just fails...

Reports don't often talk about big files testing... hope it will help other repackers!

[AHMED SAMI]

Thank You Pro

[felice2011]

How To Benchmark I took the ReflateMT version with a 122MB input.

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 09:51:24
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+reflateMT 122,1°MB 240,4°MB 196,89 % 1°Files 00:00:59:925 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mreflate
================================================== ================================================== =========================================


Here I've tested both versions "l & k" with CLS support ...

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 11:24:31
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+Refl1_K_CLS 122,1°MB 237,5°MB 194,51 % 1°Files 00:01:16:877 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mreflate
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 11:29:48
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+Refl1_L_CLS 122,1°MB 237,5°MB 194,51 % 1°Files 00:01:16:876 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mreflate
================================================== ================================================== =========================================


Between the two versions reflate with CLS "l & k" to understand, there is no substantial difference, a difference between the reflate CLS version and the version ReflateMT, in favor of the latter in terms of speed, and with a higher threshold of reflate.
The reflate version CLS will be continuously updated, hopefully soon in new improvements.

[Razor12911]

How To Benchmark I took the ReflateMT version with a 122MB input.

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 09:51:24
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+reflateMT 122,1°MB 240,4°MB 196,89 % 1°Files 00:00:59:925 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mreflate
================================================== ================================================== =========================================


Here I've tested both versions "l & k" with CLS support ...

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 11:24:31
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+Refl1_K_CLS 122,1°MB 237,5°MB 194,51 % 1°Files 00:01:16:877 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mreflate
================================================== ================================================== =========================================

-↓- [ CMD Bench.Test.Info v0.0.6c ] -↓- Compressed Archive Completed At -↓- 15/10/2016 11:29:48
================================================== ================================================== =========================================
○ [ ALGORITHM ] ○ [ INPUT SIZE ] ○ [ OUTPUT SIZE ] ○ [ RATIO.% ] ○ [ ARCHIVED FILES ] ○ [ COMP.TIME ] ○ [ SYNTAX - OPTIONS - ARGUMENTS ] ○

Arc+Refl1_L_CLS 122,1°MB 237,5°MB 194,51 % 1°Files 00:01:16:876 a -ma9 -ds -lc1024 -ld1024 -ep1 -di -i2 -ed -r -s; -w"temp" -mreflate
================================================== ================================================== =========================================


Between the two versions reflate with CLS "l & k" to understand, there is no substantial difference, a difference between the reflate CLS version and the version ReflateMT, in favor of the latter in terms of speed, and with a higher threshold of reflate.
The reflate version CLS will be continuously updated, hopefully soon in new improvements.
You're on fire with benchmarks, wanna bench something?

[Blade2239]

This is the one to use, very reliable, not sure about speed though. Great compression... nah, what you're looking for is great precompression.
[QUOTE]
Yeah forgot to type pre
But the speed is not good. Is there any way to decompress after precompressed file with 50MB/s or more..... Brewster last time I used reflate it took allott of time

[Razor12911]

50mb/s? be realistic.
Take a certain input, compress it using these settings and tell me what speed you're getting.
If the input has a lot of streams or 99% of it is covered by deflate streams, that's roughtly the speed you're going to get.

multi threading is the only way for more speed.

[Blade2239]

Ok I tried it
On hitman silent assassin's
Game size 497 mb
Used this reflate it makes this to 1.23 gb
But while compressing it with 7zip ultra, winrar with maximum, compression ratio is 90℅ why??
Then why I use this...
I tried gozarck reflate it recompress it to 1.23 gb and compressed it to 185 mb no error........

2. While decompression it starts quickly to 14 or 15℅ and stuck. Again try to decompress it extracts it. But I tried to decompress again but it stucks again same like first time.....

My arc ini
[External compressor:reflate]
header = 0
packcmd = reflate -process $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = reflate -restore - - <stdin> <stdout>

Packing batch
del /q data.arc
arc.exe a -ep1 -r -ed -w.\temp -s4096m -mreflate data.arc "pack\*"
pause

Any body

[Blade2239]

Did anyone know why it stucks on asc

[aswadd]

Ok I tried it
On hitman silent assassin's
Game size 497 mb
Used this reflate it makes this to 1.23 gb
But while compressing it with 7zip ultra, winrar with maximum, compression ratio is 90℅ why??
Then why I use this...
I tried gozarck reflate it recompress it to 1.23 gb and compressed it to 185 mb no error........

2. While decompression it starts quickly to 14 or 15℅ and stuck. Again try to decompress it extracts it. But I tried to decompress again but it stucks again same like first time.....

My arc ini
[External compressor:reflate]
header = 0
packcmd = reflate -process $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = reflate -restore - - <stdin> <stdout>

Packing batch
del /q data.arc
arc.exe a -ep1 -r -ed -w.\temp -s4096m -mreflate data.arc "pack\*"
pause

Any body

1-you shouldn't put reflate settings in arc.ini as it has its own cls ,I mentioned that in EGYcompressor
http://i.imgur.com/QP6e3dG.jpg
2- I used this reflate many times on large files (more than 2GB) & compression & decompression was success
3- try from cmd to extract the archive & see if it still stuck at 15%

[Blade2239]

1-you shouldn't put reflate settings in arc.ini as it has its own cls ,I mentioned that in EGYcompressor
http://i.imgur.com/QP6e3dG.jpg
2- I used this reflate many times on large files (more than 2GB) & compression & decompression was success
3- try from cmd to extract the archive & see if it still stuck at 15%
Yeah if I use batch file there is no problem...
But when I put this in asc in arc extract with resources all required it stuck Everytime.....

[aswadd]

Yeah if I use batch file there is no problem...
But when I put this in asc in arc extract with resources all required it stuck Everytime.....

did you try to delete reflate from arc.ini for ASC ??
if yes & the same happened try on normal inno script if the same you have to make inno launch a bat file after extracting which will unpack reflated files :(

[Lucas65]

Hello razor. I tested on .sds file of Mafia 3 (32mb). Win 7 64 bit on fx 8350.

Reflate level 1:
FreeArc 0.67 (March 15 2014) creating archive: data\data1l1.arc
Compressed 1 file, 33,397,949 => 56,319,667 bytes. Ratio 168.63%
Compression time: cpu 0.05 sec/real 14.87 sec = 0%. Speed 2.25 mB/s
All OK

Reflate level 9:
FreeArc 0.67 (March 15 2014) creating archive: data\data1l9.arc
Compressed 1 file, 33,397,949 => 50,587,917 bytes. Ratio 151.47%
Compression time: cpu 0.03 sec/real 17.82 sec = 0%. Speed 1.87 mB/s
All OK

This is ok but with level 9 it is slower and a lower ratio. Have you got an idea?
With other reflate versions there was marked how many stream found.
It seemed useful so I do an idea if the result was good.
I tried on xp 32bit but work only 64bit, pity.
Very thanks for job Razor!

[ZakirAhmad]

I compresed files successfully with above cls +reflate but i am getting extraction problems... Sometime the extraction stucks at 5 sometimes 50... any suggestions please...
Any other reflate with cls or stdin stdout available please do tell me... I also heared about reflateMt, how can i find this please... I need it badly pleease...

[COPyCAT]

Hello razor. I tested on .sds file of Mafia 3 (32mb). Win 7 64 bit on fx 8350.

Reflate level 1:
FreeArc 0.67 (March 15 2014) creating archive: data\data1l1.arc
Compressed 1 file, 33,397,949 => 56,319,667 bytes. Ratio 168.63%
Compression time: cpu 0.05 sec/real 14.87 sec = 0%. Speed 2.25 mB/s
All OK

Reflate level 9:
FreeArc 0.67 (March 15 2014) creating archive: data\data1l9.arc
Compressed 1 file, 33,397,949 => 50,587,917 bytes. Ratio 151.47%
Compression time: cpu 0.03 sec/real 17.82 sec = 0%. Speed 1.87 mB/s
All OK

This is ok but with level 9 it is slower and a lower ratio. Have you got an idea?
With other reflate versions there was marked how many stream found.
It seemed useful so I do an idea if the result was good.
I tried on xp 32bit but work only 64bit, pity.
Very thanks for job Razor!

@Lucas65, for your information the level between 1-9 isn't a compression level. In Reflate things work different from the level setting. Mostly you get better compression and time at level 6, sometime level 9 or even level 1. You must try a different level for every different kind of file. God Speed.

[chum]

I'm without luck with a specific game. Pzlib v3 just crashes during recompression or cant find deflate streams in the game. CLSreflate find the streams and recompress fine with good final ratio after srep+lzma. The problem than during decompression it fails.

Tried reflateMT but it stuck always at 59% when compressing.

[Razor12911]

This fails during decompression?

[chum]

This fails during decompression?

CLSreflate fails in decompression, yes. PZlib cant find headerless streams during compression. I think the error for reflate in decompression is a bug of reflate itself with the file type.

Game is from PS3.

ERROR: CRC failed in "unpacked\c0000\c0000_a2x.anibnd". File is broken.

If i try compressing every file from the folder individually then decompression works fine, but when i just compress all folder together with reflate it cause CRC error in decompression.

[Simorq]

CLSreflate fails in decompression, yes. PZlib cant find headerless streams during compression. I think the error for reflate in decompression is a bug of reflate itself with the file type.

Game is from PS3.

ERROR: CRC failed in "unpacked\c0000\c0000_a2x.anibnd". File is broken.

If i try compressing every file from the folder individually then decompression works fine, but when i just compress all folder together with reflate it cause CRC error in decompression.

http://krinkels.org/threads/reflate-api.3419/

[Gupta]

actually you will never get the error in PZLIb V3H if you use reflate dll and smaller value with -s
looks like the error is in razor methods(m1,m2) but not with reflate itself
try using something like pzlib:m2:x:s16:c96m:t100p
arc.ini


[External compressor:pzlib]
header = 0
packcmd = pzlib64 e {options} - -o - <stdin> <stdout>
unpackcmd = pZLib64 d -t100p - -o - <stdin> <stdout>

[chum]

What exatly -s parameter does? I not even added this but using reflate.dll worked, now pzlib detect streams and can decompress it without error. Thanks for simorq and princegupta2000.

[Simorq]

[External compressor:pZLib]
header = 0
packcmd = pZLib e -m2 -c32m -t25p - -o - <stdin> <stdout>
unpackcmd = pZLib d -t100p - -o - <stdin> <stdout>

without error

[chum]

I just noticed than if i use one sollid block with this command -s; it cause error in decompression with Pzlib+reflate.dll in this specific game i trying. If i set manual solid block size with -s8192m it decompress without error.

Simorq -t#p only dictate the number of your CPU threads in percentage, does the thread numbers used affect reliability for compression to avoid errors? When i decompress i use higher amount of threads because decompression speed with pzlib is very sensitive with number of CPU threads used.


Compressed 12,723 files, 8,971,439,002 => 3,806,018,872 bytes. Ratio 42.42%
Compression time: cpu 17.97 sec/real 2054.92 sec = 1%. Speed 4.37 mB/s
All OK

Decompression using -t6:
Extracted 12,723 files, 3,806,018,872 => 8,971,439,002 bytes. Ratio 42.42%
Extraction time: cpu 20.91 sec/real 271.14 sec = 8%. Speed 33.09 mB/s
All OK

[KaktoR]

What's the difference between k and l in archive?