ZSTD precompressor (Side project)

[Razor12911]

Hi guys, here's another tool and to be honest, I have no idea if this thing works properly, I don't have space on my PC and I don't want to start deleting stuff without proper inspection as to what is important and what isn't.

As you can see XD: 24086

Anyways enough about that, I did run a single test on the first 200mb of Resident Evil 2 of that massive pak file it has and this was the result:

Compressed 1 file, 209,715,200 => 390,911,469 bytes. Ratio 186.40%
Compression time: cpu 0.38 sec/real 10.11 sec = 4%. Speed 20.73 mB/s

Extracted 1 file, 390,911,469 => 209,715,200 bytes. Ratio 186.40%
Extraction time: cpu 0.63 sec/real 8.06 sec = 8%. Speed 26.03 mB/s

The final result is for you to find out.

[doofoo24]

i guess it will work on other games with zstd...

[Razor12911]

i guess it will work on other games with zstd...

As long as they have the necessary zstd headers for some zstd api functions to work because this is based completely on zstd functions, I didn't come up with custom functions like I did with oodle or workarounds or anything like that.

All should have the 0xfd2fb528 header, as a result. This shouldn't work on Mass Effect Andromeda because if I'm not mistake, that one has a header 0xfd2fb526 and I think different functions existed for that version.

[Mini]

Resident Evil 2 files

Compressed 10 files, 24,055,058,672 => 34,897,791,943 bytes. Ratio 145.07%
Compression time: cpu 21.17 sec/real 637.74 sec = 3%. Speed 37.72 mB/s
All OK

[KaktoR]

Mass Effect Andromeda

zstdrec
Compressed 1 file, 533,887,818 => 632,155,533 bytes. Ratio 118.41%
Compression time: cpu 0.59 sec/real 10.05 sec = 6%. Speed 53.14 mB/s
All OK

Extracted 1 file, 632,155,533 => 533,887,818 bytes. Ratio 118.41%
Extraction time: cpu 0.58 sec/real 5.64 sec = 10%. Speed 94.64 mB/s
All OK



XTool (t100p)
Compressed 1 file, 533,887,818 => 632,416,589 bytes. Ratio 118.45%
Compression time: cpu 0.50 sec/real 8.89 sec = 6%. Speed 60.05 mB/s
All OK

Extracted 1 file, 632,416,589 => 533,887,818 bytes. Ratio 118.45%
Extraction time: cpu 0.48 sec/real 7.55 sec = 6%. Speed 70.74 mB/s
All OK

Thanks razor!

[Razor12911]

Mass Effect Andromeda

zstdrec
Compressed 1 file, 533,887,818 => 632,155,533 bytes. Ratio 118.41%
Compression time: cpu 0.59 sec/real 10.05 sec = 6%. Speed 53.14 mB/s
All OK

Extracted 1 file, 632,155,533 => 533,887,818 bytes. Ratio 118.41%
Extraction time: cpu 0.58 sec/real 5.64 sec = 10%. Speed 94.64 mB/s
All OK



XTool (t100p)
Compressed 1 file, 533,887,818 => 632,416,589 bytes. Ratio 118.45%
Compression time: cpu 0.50 sec/real 8.89 sec = 6%. Speed 60.05 mB/s
All OK

Extracted 1 file, 632,416,589 => 533,887,818 bytes. Ratio 118.45%
Extraction time: cpu 0.48 sec/real 7.55 sec = 6%. Speed 70.74 mB/s
All OK

Thanks razor!

omg it works :D :D :D

Anyways, see you guys later... :)

[darkwolves]

going to repack resident evil 2 with this and do a test install

[felice2011]

Excellent Zee.

https://i.ibb.co/BwTvDSc/comp.png

https://i.ibb.co/1J6RRyV/De-comp.png

Now there is only test the whole file with the complete game.:D

[FitGirl]

Finally! Thanks, Razor12911!

[AHMED SAMI]

Thanks Razor12911

[darkwolves]

zstdrec+srep+lolz
initial size 23 gb
repack size 15.7 gb
only ripped wallpapers
unpacks in 20 minutes on my pc
no errors unpacked on the first try and the game plays normally

[nik1967]

Razor12911, thank you very much!
Like the above wrote
zstdrec+srep+lolz
initial size 23,3 gb
repack size 15.7 gb
unpacks in 13:07 minutes on my pc
no errors unpacked on the first try and the game plays normally
MD5 files are identical

[panker1992]

any ideas how to unpack game files and pack them again ??

[ZAZA4EVER]

Thanks Very Much Razor for Amazing Tool

[GTX590]

Any chance that it works with Forza Horizon 4?

[elit]

Thank you as well and its great to see you kicking again. If I may ask, why not just update zstd inside xtool and make another separate one?

[darkwolves]

i've been collecting the side projects i sometimes tend to use them over Xtool

[darkwolves]

any ideas how to unpack game files and pack them again ??

this was passed onto me by KaKtor it extracts it
but i have yet to successfully pack it back up

http://forum.xentax.com/viewtopic.php?f=10&t=19304

[Razor12911]

Any chance that it works with Forza Horizon 4?

Am not sure lol

Thank you as well and its great to see you kicking again. If I may ask, why not just update zstd inside xtool and make another separate one?

because xtool is in a horrible state currently, a lot of months passed and I just decided to open the source recently and it didn't seem like I was the one who the WTF code it contains :D

So I decided start smaller projects to see how those fare in what I plan to do in future.

[elit]

because xtool is in a horrible state currently, a lot of months passed and I just decided to open the source recently and it didn't seem like I was the one who the WTF code it contains :D


I know this feeling. Coded my own "Total Commander" in Borland C++ years ago and later decided to revisit. My code is actually extremely clean and ordered(others who saw it said as well), yet I gave up as it would take too much effort/time to digest it again.

I suspect this have to do with object oriented languages once they grow in size, never had such problem with C projects and even reading others work (like before mentioned LZ4 or plzip which I forked to xnlz) was such a breeze. Yet I could never properly digest C++ code not only from others, but even my own - well organized. And I used to be really good at C++ once before(before I forgot most of that crap) including complete STL knowledge, all kind of tricks etc..

Really, just drop that crap called "delphi pascal" and try plain C instead ;). I am almost certain you will not regret it + you will gain speed as well and besides, most of your tools are command line without GUI anyway :o.

[Razor12911]

I know this feeling. Coded my own "Total Commander" in Borland C++ years ago and later decided to revisit. My code is actually extremely clean and ordered(others who saw it said as well), yet I gave up as it would take too much effort/time to digest it again.

I suspect this have to do with object oriented languages once they grow in size, never had such problem with C projects and even reading others work (like before mentioned LZ4 or plzip which I forked to xnlz) was such a breeze. Yet I could never properly digest C++ code not only from others, but even my own - well organized. And I used to be really good at C++ once before(before I forgot most of that crap) including complete STL knowledge, all kind of tricks etc..

Really, just drop that crap called "delphi pascal" and try plain C instead ;). I am almost certain you will not regret it + you will gain speed as well and besides, most of your tools are command line without GUI anyway :o.

I only code for fun bro or I should I say I used to as it is not fun anymore :/, so learning how to write in another language would simply mean I'm not doing it as a hobby but now taking a path towards something I do as a profession :(

[L0v3craft]

Thank you Razor, btw here it doesn't pump(pre-compress) the files of the folders: 920560, 920561, 920562, 920563, 920564, 920567, 920568, 920569, 920570. Is the normal ? Btw it works with re_chunk_000.pak (22.1GB).

[FitGirl]

Thank you Razor, btw here it doesn't pump(pre-compress) the files of the folders: 920560, 920561, 920562, 920563, 920564, 920567, 920568, 920569, 920570. Is the normal ? Btw it works with re_chunk_000.pak (22.1GB).
DLC paks are compressed with deflate

[L0v3craft]

Edit: sorry you can delete this.

[yasitha]

I have try it on Resident Evil 7 won't work :(

I know Re7 had zlib but i think something else also inthere like lzo, lz4 or zstd different header.. Maybe..

[ZAZA4EVER]

Thanks Razor ... Tool Works Fine ...

ZSTDREC

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1zrec.bin
Compressed 1 file, 33,451,416,458 => 49,631,422,519 bytes. Ratio 148.37%
Compression time: cpu 95.28 sec/real 1752.97 sec = 5%. Speed 19.08 mB/s
All OK


Xtool V0.9 Zlib

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1xtool.bin
Compressed 1 file, 33,451,416,458 => 34,951,267,935 bytes. Ratio 104.48%
Compression time: cpu 94.34 sec/real 1402.57 sec = 7%. Speed 23.85 mB/s
All OK

[yasitha]

Thanks Razor ... Tool Works Fine ...

ZSTDREC

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1zrec.bin
Compressed 1 file, 33,451,416,458 => 49,631,422,519 bytes. Ratio 148.37%
Compression time: cpu 95.28 sec/real 1752.97 sec = 5%. Speed 19.08 mB/s
All OK


Xtool V0.9 Zlib

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1xtool.bin
Compressed 1 file, 33,451,416,458 => 34,951,267,935 bytes. Ratio 104.48%
Compression time: cpu 94.34 sec/real 1402.57 sec = 7%. Speed 23.85 mB/s
All OK



Awesome!!! GOOD JOB

so game had both,
like Re2 zlib+zstd..

Unfortunately i don't have The Game.
Waiting for it to happen :D

[observer23232]

Thanks Razor!

BTW:
im used taskkill for zstdrec.exe

function IsProcessRunning(FileName: String): Boolean;
var
objSWbemLocator, objSWbemServices: Variant;
begin
try
objSWbemLocator:= CreateOleObject('WbemScripting.SWbemLocator');
except
ShowExceptionMessage;
Exit;
end;
objSWbemServices:= objSWbemLocator.ConnectServer();
objSWbemServices.Security_.ImpersonationLevel := 3;
Result := (objSWbemServices.ExecQuery('SELECT * FROM Win32_Process WHERE Name="' + FileName + '"').Count > 0);
end;

Cancel Button or ISDoneError
var
#ifdef zstdrec
ResC: Integer;
#endif
begin
------
#ifdef zstdrec
if IsProcessRunning('zstdrec.exe') then
Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM zstdrec.exe', '', SW_HIDE, ewWaitUntilTerminated, ResC);
#endif
end;

[prince55]

Does zstdrec work with resident evil 3 remake ?
hmm , it does not expand for me !!
Are game files encrypted ?

[panker1992]

you can check on my guide :D on RE3

[prince55]

you can check on my guide :D on RE3

Where !?:)

[Grumpy]

Where !?:)

HERE (https://fileforums.com/showpost.php?p=485160&postcount=1622)

[ZAZA4EVER]

Xtool 0.9 works fine on Re3 with big size 400mb from Panker1992

[darkwolves]

nope zlib streams for re3 dunno why they switched streams for but they did