FileForums - ZSTD precompressor (Side project)

FileForums (https://fileforums.com/index.php)

- Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)

- - ZSTD precompressor (Side project) (https://fileforums.com/showthread.php?t=102599)

Razor12911

09-02-2019 08:27

ZSTD precompressor (Side project)

2 Attachment(s)

Hi guys, here's another tool and to be honest, I have no idea if this thing works properly, I don't have space on my PC and I don't want to start deleting stuff without proper inspection as to what is important and what isn't.

As you can see XD: Attachment 24086

Anyways enough about that, I did run a single test on the first 200mb of Resident Evil 2 of that massive pak file it has and this was the result:

Code:

Compressed 1 file, 209,715,200 => 390,911,469 bytes. Ratio 186.40%

Compression time: cpu 0.38 sec/real 10.11 sec = 4%. Speed 20.73 mB/s



Extracted 1 file, 390,911,469 => 209,715,200 bytes. Ratio 186.40%

Extraction time: cpu 0.63 sec/real 8.06 sec = 8%. Speed 26.03 mB/s

The final result is for you to find out.

doofoo24

09-02-2019 08:29

i guess it will work on other games with zstd...

Razor12911

09-02-2019 08:35

Quote:

Originally Posted by doofoo24 (Post 479419)

i guess it will work on other games with zstd...

As long as they have the necessary zstd headers for some zstd api functions to work because this is based completely on zstd functions, I didn't come up with custom functions like I did with oodle or workarounds or anything like that.

All should have the 0xfd2fb528 header, as a result. This shouldn't work on Mass Effect Andromeda because if I'm not mistake, that one has a header 0xfd2fb526 and I think different functions existed for that version.

Mini	09-02-2019 08:51

Resident Evil 2 files

Compressed 10 files, 24,055,058,672 => 34,897,791,943 bytes. Ratio 145.07%
Compression time: cpu 21.17 sec/real 637.74 sec = 3%. Speed 37.72 mB/s
All OK

KaktoR

09-02-2019 08:57

Mass Effect Andromeda

Code:

zstdrec

Compressed 1 file, 533,887,818 => 632,155,533 bytes. Ratio 118.41%

Compression time: cpu 0.59 sec/real 10.05 sec = 6%. Speed 53.14 mB/s

All OK



Extracted 1 file, 632,155,533 => 533,887,818 bytes. Ratio 118.41%

Extraction time: cpu 0.58 sec/real 5.64 sec = 10%. Speed 94.64 mB/s

All OK







XTool (t100p)

Compressed 1 file, 533,887,818 => 632,416,589 bytes. Ratio 118.45%

Compression time: cpu 0.50 sec/real 8.89 sec = 6%. Speed 60.05 mB/s

All OK



Extracted 1 file, 632,416,589 => 533,887,818 bytes. Ratio 118.45%

Extraction time: cpu 0.48 sec/real 7.55 sec = 6%. Speed 70.74 mB/s

All OK

Thanks razor!

Razor12911

09-02-2019 09:00

Quote:

Originally Posted by KaktoR (Post 479422)

Mass Effect Andromeda

Code:

zstdrec

Compressed 1 file, 533,887,818 => 632,155,533 bytes. Ratio 118.41%

Compression time: cpu 0.59 sec/real 10.05 sec = 6%. Speed 53.14 mB/s

All OK



Extracted 1 file, 632,155,533 => 533,887,818 bytes. Ratio 118.41%

Extraction time: cpu 0.58 sec/real 5.64 sec = 10%. Speed 94.64 mB/s

All OK







XTool (t100p)

Compressed 1 file, 533,887,818 => 632,416,589 bytes. Ratio 118.45%

Compression time: cpu 0.50 sec/real 8.89 sec = 6%. Speed 60.05 mB/s

All OK



Extracted 1 file, 632,416,589 => 533,887,818 bytes. Ratio 118.45%

Extraction time: cpu 0.48 sec/real 7.55 sec = 6%. Speed 70.74 mB/s

All OK

Thanks razor!

omg it works :D :D :D

Anyways, see you guys later... :)

darkwolves

09-02-2019 09:40

going to repack resident evil 2 with this and do a test install

felice2011

09-02-2019 12:29

Excellent Zee.

https://i.ibb.co/BwTvDSc/comp.png

https://i.ibb.co/1J6RRyV/De-comp.png

Now there is only test the whole file with the complete game.:D

FitGirl

09-02-2019 13:33

Finally! Thanks, Razor12911!

AHMED SAMI

09-02-2019 14:25

Thanks Razor12911

darkwolves

09-02-2019 19:01

Resident evil 2 Remake

zstdrec+srep+lolz
initial size 23 gb
repack size 15.7 gb
only ripped wallpapers
unpacks in 20 minutes on my pc
no errors unpacked on the first try and the game plays normally

nik1967

10-02-2019 10:24

Razor12911, thank you very much!
Like the above wrote
zstdrec+srep+lolz
initial size 23,3 gb
repack size 15.7 gb
unpacks in 13:07 minutes on my pc
no errors unpacked on the first try and the game plays normally
MD5 files are identical

panker1992

10-02-2019 11:29

any ideas how to unpack game files and pack them again ??

ZAZA4EVER

10-02-2019 12:10

Thanks Very Much Razor for Amazing Tool

GTX590

10-02-2019 13:30

Any chance that it works with Forza Horizon 4?

elit	10-02-2019 14:05

Thank you as well and its great to see you kicking again. If I may ask, why not just update zstd inside xtool and make another separate one?

darkwolves

10-02-2019 18:43

i've been collecting the side projects i sometimes tend to use them over Xtool

darkwolves

10-02-2019 18:52

Quote:

Originally Posted by panker1992 (Post 479446)

any ideas how to unpack game files and pack them again ??

this was passed onto me by KaKtor it extracts it
but i have yet to successfully pack it back up

http://forum.xentax.com/viewtopic.php?f=10&t=19304

Razor12911

10-02-2019 20:29

Quote:

Originally Posted by GTX590 (Post 479449)

Any chance that it works with Forza Horizon 4?

Am not sure lol

Quote:

Originally Posted by elit (Post 479451)

Thank you as well and its great to see you kicking again. If I may ask, why not just update zstd inside xtool and make another separate one?

because xtool is in a horrible state currently, a lot of months passed and I just decided to open the source recently and it didn't seem like I was the one who the WTF code it contains :D

So I decided start smaller projects to see how those fare in what I plan to do in future.

elit	11-02-2019 05:19

Quote:

Originally Posted by Razor12911 (Post 479458)

because xtool is in a horrible state currently, a lot of months passed and I just decided to open the source recently and it didn't seem like I was the one who the WTF code it contains :D

I know this feeling. Coded my own "Total Commander" in Borland C++ years ago and later decided to revisit. My code is actually extremely clean and ordered(others who saw it said as well), yet I gave up as it would take too much effort/time to digest it again.

I suspect this have to do with object oriented languages once they grow in size, never had such problem with C projects and even reading others work (like before mentioned LZ4 or plzip which I forked to xnlz) was such a breeze. Yet I could never properly digest C++ code not only from others, but even my own - well organized. And I used to be really good at C++ once before(before I forgot most of that crap) including complete STL knowledge, all kind of tricks etc..

Really, just drop that crap called "delphi pascal" and try plain C instead ;). I am almost certain you will not regret it + you will gain speed as well and besides, most of your tools are command line without GUI anyway :o.

Razor12911

13-02-2019 17:27

Quote:

Originally Posted by elit (Post 479465)

I only code for fun bro or I should I say I used to as it is not fun anymore :/, so learning how to write in another language would simply mean I'm not doing it as a hobby but now taking a path towards something I do as a profession :(

L0v3craft

17-02-2019 08:05

Thank you Razor, btw here it doesn't pump(pre-compress) the files of the folders: 920560, 920561, 920562, 920563, 920564, 920567, 920568, 920569, 920570. Is the normal ? Btw it works with re_chunk_000.pak (22.1GB).

FitGirl

17-02-2019 22:06

Quote:

Originally Posted by L0v3craft (Post 479571)

DLC paks are compressed with deflate

L0v3craft

23-02-2019 11:13

Edit: sorry you can delete this.

yasitha

05-03-2019 08:11

I have try it on Resident Evil 7 won't work :(

I know Re7 had zlib but i think something else also inthere like lzo, lz4 or zstd different header.. Maybe..

ZAZA4EVER

08-03-2019 00:24

Devil May Cry 5

Thanks Razor ... Tool Works Fine ...

ZSTDREC

Code:

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1zrec.bin

Compressed 1 file, 33,451,416,458 => 49,631,422,519 bytes. Ratio 148.37%

Compression time: cpu 95.28 sec/real 1752.97 sec = 5%. Speed 19.08 mB/s

All OK

Xtool V0.9 Zlib

Code:

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1xtool.bin

Compressed 1 file, 33,451,416,458 => 34,951,267,935 bytes. Ratio 104.48%

Compression time: cpu 94.34 sec/real 1402.57 sec = 7%. Speed 23.85 mB/s

All OK

yasitha

08-03-2019 00:35

Quote:

Originally Posted by ZAZA4EVER (Post 480065)

Thanks Razor ... Tool Works Fine ...

ZSTDREC

Code:

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1zrec.bin

Compressed 1 file, 33,451,416,458 => 49,631,422,519 bytes. Ratio 148.37%

Compression time: cpu 95.28 sec/real 1752.97 sec = 5%. Speed 19.08 mB/s

All OK

Xtool V0.9 Zlib

Code:

FreeArc 0.67 (March 15 2014) creating archive: ZAZA.Data1xtool.bin

Compressed 1 file, 33,451,416,458 => 34,951,267,935 bytes. Ratio 104.48%

Compression time: cpu 94.34 sec/real 1402.57 sec = 7%. Speed 23.85 mB/s

All OK

Awesome!!! GOOD JOB

so game had both,
like Re2 zlib+zstd..

Unfortunately i don't have The Game.
Waiting for it to happen :D

observer23232

04-05-2019 02:16

Thanks Razor!

BTW:
im used taskkill for zstdrec.exe

Code:

function IsProcessRunning(FileName: String): Boolean;

var

  objSWbemLocator, objSWbemServices: Variant;

begin

  try

    objSWbemLocator:= CreateOleObject('WbemScripting.SWbemLocator');

  except

    ShowExceptionMessage;

    Exit;

  end;

  objSWbemServices:= objSWbemLocator.ConnectServer();

  objSWbemServices.Security_.ImpersonationLevel := 3;

  Result := (objSWbemServices.ExecQuery('SELECT * FROM Win32_Process WHERE Name="' + FileName + '"').Count > 0);

end;

Cancel Button or ISDoneError

Code:

var

#ifdef zstdrec

  ResC: Integer;

#endif  

begin

------  

    #ifdef zstdrec

      if IsProcessRunning('zstdrec.exe') then

       Exec(ExpandConstant('{sys}\taskkill.exe'), ' /F /IM zstdrec.exe', '', SW_HIDE, ewWaitUntilTerminated, ResC);

    #endif

end;

prince55

03-04-2020 19:50

Does zstdrec work with resident evil 3 remake ?
hmm , it does not expand for me !!
Are game files encrypted ?

panker1992

06-04-2020 17:30

you can check on my guide :D on RE3

prince55

06-04-2020 21:18

Quote:

Originally Posted by panker1992 (Post 485161)

you can check on my guide :D on RE3

Where !?:)

Grumpy

07-04-2020 06:04

Quote:

Originally Posted by prince55 (Post 485162)

Where !?:)

HERE

ZAZA4EVER

10-04-2020 13:33

Xtool 0.9 works fine on Re3 with big size 400mb from Panker1992

darkwolves

04-05-2020 00:00

nope zlib streams for re3 dunno why they switched streams for but they did

All times are GMT -7. The time now is 14:47.