Fast Brute (test)
 

Like the title says, it's a test.

What needs testing here is comparison with what reflate does and/or what precomp does.

I noticed a bug after compiling that when you compress the output of brute, you get a very small output
Well it's a false output so it's just a waste of time if you want to see how much you could get if this
program as working. I was going to ask someone to try this program on Mad Max and maybe get size which
is less than 4GB in less than a day, unlike precomp (brute), this is faster though it achieves 0.2mb/s
on data without zlibs (tested on audio files) and can push up to 40mb/s or more on an input with
a lot of zlib streams (Well it is 0.2mb/s on my 9 year old Core 2 Extreme QX9650).

I already know that it doesn't work just as yet which is why I said just compare final size
with another zlib recompressor. (I edited zlib source to improve speed and that is what happened)
I get false data or repeated data making lzma compress the whole thing to a couple of kilos
and get 1% ratio which is false but working it though to be able to get correct data and make this work.


What to test in comparison with reflate:
The output size, since some people say "reflate" is the only recompressor that can detect "invalid" streams
and others can't, well you could try this program. Again, if an stream is invalid, its invalid, there is
nothing further you can do, but as I said detection and restoration are the only factors that separate these
zlib recompressors. (You know yourself)

You can also test on GTAV, I got expansion on one of the rpf files meaning it does work on GTAV as well.


What to test in comparison with precomp:
The output size and speed.

Files that you test on:
Make sure they do have zlib streams else you'll get 0.2mb/s. (brute scanning speed)

If you got Mad Max, run the program on the game, don't apply srep or lzma and those other things,
if you know what files reflate doesn't work on but precomp (brute) works, try running this.

A test on a benchmarking file that I always use to test everything is the UI.sb file and here are the results:
Input : 82.3MB
// Bigger is better
Brute (Test) : 421MB (Without verification)
Precomp + Reflate : 421MB (Evidence that reflate isn't really brute, it had to work with precomp to get that size)
Precomp (Brute) : 421MB (Probably, I had to cancel it, it takes long)
Precomp (Intense) : 373MB
Reflate (Gozarck) : 361MB
pZlib : 348MB

In terms of speed while working on a file (UI.sb):
Input : 82.3MB
Slowest to fastest:
pZlib : 348MB (6.39 seconds)
Brute (Test) : 421MB (47.70 seconds) //Quite slow but working on it.

Work of Razor12911 and Panker1992

GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 308 MB (2 min)
Brute+Srep+lzma:max: 55 MB (Monster Compressed :eek:)

Reflate u3 : 312 MB (10 min)
x64b.rpf (191MB) x64b.ref (359MB) x64b.srep (253MB) x64b.arc (162MB)

Brute (Test)
http://uupload.ir/files/9skb_brute.jpg

Reflate
http://uupload.ir/files/8tlp_reflate.jpg

Good

simorg it's very good but which lzma code did you use on reflate u3? and which lzma method is the best lzma:max or another long method (using it in masked-ms).and what is the best compressor for after reflate? lzma nanozip fast paq8 (fp8) zpaq..

Mad Max

game24.arc - 812 mb

data.arc - 1.45 gb

time ~ 5 min

Have to work on that 4mb difference
Like I said. The result you'll be getting are false after applying srep and lzma, bug caused after I played a little with zlib source so basically, the output size is the only thing that tells us it could work at some point.

unfortunately i give exams, and i havent had the time to test myself :D

hope you guys give some feedback, and help along the line :D

make sure to test and provide any feedback or glitches this soft makes ... keep an eye on ram and IO... :D

thanks Zee :D

Is it usable for decompression? i'm getting wclose: invalid argument (Bad file descriptor)
ERROR: CRC failed error.But veery good file size in (coh2 files) 2gb to 3gb.then srep lzma 146mb! i think something is wrong in brute.exe.with reflate 2gb to 3gb but then srep lzma 1100mb?

I wrote up there
also

Really impresive reasult razor. I was made some test and the results are very close with reflate but brute test with 1 core is more faster than reflate and the results are very close. Also some files that precomp doesn't detect in faster mode is detected by brute and reflate too. Very nice. I request the decode! For more test.

Razor. if you build some 'cls_brute.dll' will work with ISDone for extract files. :D

Told ya, detection and restoration.

I still have to sort out restoration before I can make decode.

Test 2 uploaded.

You can now apply srep and lzma to get theoretical output size, fixed the so called bug.

Plus, I added Multi threading, I just want to see how much speed can I get out of this thing before I even start with restoration. Multi threading starts working if you got an input more than 64MB, It's 1 Thread = 64MB, watch RAM usage and CPU Usage, it uses the number of threads you have minus 1. So if you got an i7, it will use 7 threads which is 7 x 64MB =448MB, so expect +-1536MB RAM usage, but if you got crazy inputs like that game called Dirt Rally, then you're going to need about 6GB RAM.

You can go again, I want to see what will happen to that 5 minutes you got. (Hopefully it is going to be less)

@Everyone
I'm taking a knee, Panker is busy with exams, but you can keep testing.

multi-th log
data.arc (1 th) - 1.45 gb
DATA.ARC (multi-th) - 1.49 GB

time (1 th) - ~10 min
time (multi) - ~6 min

GTA V x64b.rpf
Input : 142 MB
Brute (Test) : 309 MB (2.5 min)
Reflate Bilawal (Refl8v100): 312 MB (3 min)
Brute+Srep+lzma:max: 115 MB (3.5 min)
Reflate+Srep+lzma:max: 116 MB (4 min)

Brute (Test)
Brute+Srep+lzma
GOOD:)

Thanks for very good job. I tested first brute with this results:

On file GTAV (x64a.rpf - 46.4mb)
Only brute
Compressed 1 file, 48,699,392 => 154,259,682 bytes. Ratio 316.76%
Compression time: cpu 0.06 sec/real 33.15 sec = 0%. Speed 1.47 mB/s
only Reflate
Compressed 1 file, 48,699,392 => 155,496,915 bytes. Ratio 319.30%
Compression time: cpu 0.05 sec/real 266.69 sec = 0%. Speed 0.18 mB/s

On Fifa 15 file (data_graphic1.big - 17.9 mb)
Only precomp brute
Compressed 1 file, 18,855,627 => 58,149,544 bytes. Ratio 308.39%
Compression time: cpu 0.02 sec/real 167.31 sec = 0%. Speed 0.11 mB/s
Only brute
Compressed 1 file, 18,855,627 => 58,142,118 bytes. Ratio 308.35%
Compression time: cpu 0.00 sec/real 1.05 sec = 0%. Speed 18.04 mB/s
Only reflate
Compressed 1 file, 18,855,627 => 59,147,086 bytes. Ratio 313.68%
Compression time: cpu 0.02 sec/real 234.76 sec = 0%. Speed 0.08 mB/s

I tried brute2. Thank again Razor!

"Stack space overflow: current size 80000000 bytes.
Use `+RTS -Ksize' to increase it."
what i can do with this error?

How did you get that?

I was using it with company of heroes 2 sga files (2gb) (they are compatible with reflate or any recompressor) i didn't change any files, i've enough space on my drives
edit:i splitted files now it's working.
i think it's not working with big file sizes

And how many threads does your CPU have and how much ram do you have?

8gb ram i5 4210H 4 core windows 10 64 bit

hmm. checking problem.

Test Result of game wwe16 >pac>ch>1st file
Brute:
Compressed 1 file, 32,450,560 => 54,980,006 bytes. Ratio 169.43%
Compression time: cpu 0.02 sec/real 0.80 sec = 2%. Speed 40.73 mB/s

Reflate :
Compressed 1 file, 32,450,560 => 55,593,271 bytes. Ratio 171.32%
Compression time: cpu 0.03 sec/real 10.69 sec = 0%. Speed 3.04 mB/s

pzlib :
Compressed 1 file, 32,450,560 => 32,670,323 bytes. Ratio 100.68%
Compression time: cpu 0.06 sec/real 1.84 sec = 3%. Speed 17.60 mB/s

zlib Checker by Razor :
Found streams: 14
Duration: 344 ms
30.9 MB > 31.8 MB

MT is very powerfull use every piece of ram. After to make 1 test the second test i need to free my ram. So excellent.

Extraction working!
 

Ok, I just thought, lets try if it will extract fine or not, so i taken a file from The Sims 4 game - ClientFullBuild0.package
And i got that it extracted the files but I got CRC Check Failed in the end, but the files extracted successfully, but as it is multithread, so I have only dual core processor, so it uses only 1 thread from my pc and as razor said 64mb at a time, so it decompress 64mb and then extract it and then again extract 64mb and so on..

and here, how i made it works -

I tried this with masked unpacking section v2.5.
1. Copy brute.exe from fast brute folder and paste/replace it in unpacking section folder
2. go to arc.ini of unpacking section and paste/replace this -

[External compressor:brute]
header = 0
packcmd = brute <stdin> <stdout>
unpackcmd = brute <stdin> <stdout>



Now, i am going to see if really crc or md5 is different or not with different checking tools ;)

EDIT:
I checked both files with QuickSFV, and CRC is Mismatched, I hope Razor will fix this issue!

Dude, it ain't working.

Maybe the file is filled with zero

Is anyone have method or any tips for repacking Mad Max game with brute?? Thanks for help

File is filled with actual decompressed data, I just haven't worked on restoration code.

1. Panker is busy with exams
2. I'm busy with assignments and practicals
3. I'm lazy to write decompression code :p

Use precomp 0.38 -brute0 -v -f for mad max

Actually I was asking for brute, that command I know with precomp 038, but I want to know with brute

Now I'm testing the result of "brute" with Mad Max.
Beyond to infinity of time which is putting, I'm curious to know the compression ratio.:rolleyes:

Good luck.

With the hope that has all the fallen teeth ...;)

☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢ ☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢☢

Test executed of the game "Mad Max" File "game38.arc", Tool used in compression "M&S Kompressor".

Input File = 554MB

T1 = precomp0.38+msc+srep64+dispack070+nz / Output = 419.154MB / RC = 75.68% / T°= ~ 00h.03m.50s

T2 = pzlib+msc+srep64+dispack070+nz / Output = 419.728MB / RC = 75.79% / T°= ~ 00h.02m.30s

T3 = brute+msc+srep64+dispack070+nz / Output = 282.384MB / RC = 50.99% / T°= ~ 01h.54m.16s

Truly amazing the compression ratio with the new brute compressor, but time seems exaggerated with just 554MB of tests.

Razor & Panker are doing a great job, are having my admiration. ٩(๏̯͡๏)۶

2 hours mate?! I get 0.2mb/s on data without zlibs, always like that. 554 / 0.2 / 60 = 46.167 minutes, but there is room for improvement.

I forgot to read the speed of the closing of the compression with brute, but the result is to, it would be good thing to be able to refine adding a compression level to be chosen, for example from L1 to L9, actually also the compression seems excessive we speak of 50% Mad Max is 32GB would become 16GB, I guess to install it and decompress the archive it will take two days..:D

Haha, level setting is not needed for this, it's not based on raw2hif or reflate for that matter.

A real disadvantage is compiled in this way, the compression time is too high, and certainly also the decompression time, you have to find the right balance for all, and give the user the ability to choose its own level of compression, in this way "brute" will not make a long way, I'm sorry mate.

Haha, compression time high, have you tried precomp brute? How does it compare to this with speed? BTW, it doesn't mean if compression took long, decompression will take long as well.

My interest is the new brute, I have no interest to compare the precomp with brute, personally think that the only precomp compression worthy is the "0.38" all other versions continue to generate errors on most files at the end of data compression, I tested the last 0.45 by schnaader, and personally I still do not see any substantial difference.
Ha ha...I hope it is as you say, but I doubt it:rolleyes:

Dude. "give the user the ability to choose its own level of compression"
This doesn't make sense, how can you add level for decompression? Brute is not reflate or made based on raw2hif, level setting is something that reflate does itself, brute on the other hand is search every byte then decompress.imagine a 794mb file, it has 832 618 624 bytes, brute must use trial and error for every single of those bytes starting from 1 to file size, brute checks about 200000 bytes per second on my cpu, I dont think that is slow.