Best Compression Methods for 'Specific' Games. Q&A

[L33THAK0R]

Originally Posted by panker1992

Resident Evil 8 Village

Sounds - All
Movies - Removed Optional how its made stuff
and tex files left as is Total Compressed Size = 18 GB

Blocks

3317383863 3731856949 1.wmv optional
1483173675 1648366619 2.wmv
3805438255 3826442429 3.wmv
3826442430 4070125575 4.wmv optional
4408129950 4648733769 5.wmv optional
4677339017 4892061781 6.wmv
4215713582 4408129949 7.wmv optional
4070125576 4215713581 8.wmv optional
1648366620 1773564442 9.wmv
3731856950 3805438254 10.wmv
4648733770 4677339016 11.wmv
4892061782 4930489090 12.wmv
1464006109 1470603909 13.wmv
1470603910 1483173674 14.wmv
4930489091 4932102837 15.wmv


Re-Encode High Quality = 8 MBPS Video
Re-Encode Medium Quality = 6 MBPS Video
Re-Encode Low Quality = 4 MBPS Video

sfk partcopy 1.bin -allfrom 0 re_chunk_000.pak 3317383863 -yes
sfk partcopy 2.bin -allfrom 0 re_chunk_000.pak 1483173675 -yes
sfk partcopy 3.bin -allfrom 0 re_chunk_000.pak 3805438255 -yes
sfk partcopy 4.bin -allfrom 0 re_chunk_000.pak 3826442430 -yes
sfk partcopy 5.bin -allfrom 0 re_chunk_000.pak 4408129950 -yes
sfk partcopy 6.bin -allfrom 0 re_chunk_000.pak 4677339017 -yes
sfk partcopy 7.bin -allfrom 0 re_chunk_000.pak 4215713582 -yes
sfk partcopy 8.bin -allfrom 0 re_chunk_000.pak 4070125576 -yes
sfk partcopy 9.bin -allfrom 0 re_chunk_000.pak 1648366620 -yes
sfk partcopy 10.bin -allfrom 0 re_chunk_000.pak 3731856950 -yes
sfk partcopy 11.bin -allfrom 0 re_chunk_000.pak 4648733770 -yes
sfk partcopy 12.bin -allfrom 0 re_chunk_000.pak 4892061782 -yes
sfk partcopy 13.bin -allfrom 0 re_chunk_000.pak 1464006109 -yes
sfk partcopy 14.bin -allfrom 0 re_chunk_000.pak 1470603910 -yes
sfk partcopy 15.bin -allfrom 0 re_chunk_000.pak 4930489091 -yes

This is a really silly question but how do you find/calculate the blocks for a desired file? I'm trying to accomplish a similar feat with the earlier "Far Cry" titles & "The Settlers VII (7)", regarding their Bink videos.

[:( Sad8669]

I will try RE8 as soon as i get my hand on the game files.

panker1992 you did the same thing with RE3 and also managed to compress it effeciently.

Would love to see it as low as possible.

[panker1992]

>Leethakor
I have demonstarted in the forum how to calculate and find the files and remove them!!
I have also posted a video on youtube personally that explains it with alo of details

> Sad8669
I am fixing Sound stuff from the game i am almost done.

I have managed to strip down many titles over my free time
Resident Evil 2 = 13.6 GB
Call of Cthulhu = 7.3 GB
Little Hope = 18.8GB same as Fitgirl but mine is better because the decompressed size from 55 GB is 35GB and yes its playable
Man of Medan = 17.8 GB same as above game is 30% down for decompressed size instead of 50GB its 30 and playable!!!
Biomuntant = 6.4GB this game is 25+ GB decompressed mine is 12GB decompressed and yes its playable!!!

[:( Sad8669]

Can you look into Yakuza 0 in your free time?

Fitgirl's version is : 11.7 GB
Mine is : 14.0 GB

The point is that, i know what Fitgirl has done to make it this small. She unpacked .par files which i also did and seen good results, but unlike sfk there isn't any tool to patch it back into the .par file to its original size.

It's just other information (if not discussed previously).

[panker1992]

The Total Compressed game is estimated to be 16GB no more !! will post once i pack it though
Completely Remove languages this game has total of 9 spoken languages so.
Also removing large portions of uncompressible files help with lolz - lolzi compression!!
Good Luck!!
PS. Take Extreme caution here can break the game if you dont know what you are doing - this is master dissection at it's best !!!



Movie - Blocks
3317383863 3731856949 1.wmv optional
1483173675 1648366619 2.wmv
3805438255 3826442429 3.wmv
3826442430 4070125575 4.wmv optional
4408129950 4648733769 5.wmv optional
4677339017 4892061781 6.wmv
4215713582 4408129949 7.wmv optional
4070125576 4215713581 8.wmv optional
1648366620 1773564442 9.wmv
3731856950 3805438254 10.wmv
4648733770 4677339016 11.wmv
4892061782 4930489090 12.wmv
1464006109 1470603909 13.wmv
1470603910 1483173674 14.wmv
4930489091 4932102837 15.wmv

BGM Ingame Language Audio

28717571425 28736453394
28717571425 em1000_v.bnk.2.x64.en

25315697863 25337249337
25315697863 em1260_v.bnk.2.x64.en

28671792013 28692171487
28671792013 em1261_v.bnk.2.x64.en

28692171883 28712379369
28692171883 em1262_v.bnk.2.x64.en

27929954843 27951900137
27929954843 em1310_v.bnk.2.x64.en

25834788055 25866260102
25834788055 pl1000_v.bnk.2.x64.en

27959817454 27986571026
27959817454 pl2000_v.bnk.2.x64.en

28789034961 28801457854
28789034961 em1302_v_effort.bnk.2.x64.en

BGM - Blocks - Around 700 MB of Music


Block 1 500MB File
5578232732 6096544709 bgm_resident.pck.3.x64
Block 2
807697467 829232497 bgm_em_c22.pck.3.x64
Block 3
829232498 841520221 bgm_em_c23.pck.3.x64
Block 4
871463739 891591317 bgm_em_c24.pck.3.x64
Block 5
900402239 935723696 bgm_em_c27.pck.3.x64
Block 6
1008712931 1040885578 bgm_em_c32.pck.3.x64
Block 7
470684396 530322314 movie_chapter3.pck.3.x64
Block 8
3291738440 3317383862 movie_bonus.pck.3.x64 - Optional?
Block 9
3234287977 3248181923 movie_chapter1.pck.3.x64

Spoken language - ENGLISH ONLY


721910965 807697466
731727462 em1000_v_dialogue.pck.3.x64.en

2422899086 2493439908
2422899086 em1110_v_dialogue.pck.3.x64.en

18114998188 18152135580
18119034210 em1150_v_dialogue.pck.3.x64.en

20249490548 20294360582
20254537491 em1230_v_dialogue.pck.3.x64.en

19302022331 19349746976
19302022331 em1260_v_dialogue.pck.3.x64.en

5506437541 5556561515
5506437541 em1262_v_dialogue.pck.3.x64.en

841520222 871463738
841520222 em1300_v_dialogue.pck.3.x64.en

20294360583 20341650467
20294360583 em1302_v_dialogue.pck.3.x64.en

18394439764 em1310_v_dialogue.pck.3.x64.en
18394439764 18438588281

6100084475 6599324197
6100084475 pl1000_dialogue.pck.3.x64.en

156083303 470684395
191024214 movie_chapter3_dialog.pck.3.x64.en

20354417983 20417554742
20361475021 em1320_v_dialogue.pck.3.x64.en

18251704719 em4070_v_dialogue.pck.3.x64.en
18251704719 18380860641

18454440446 18491950561
18454440446 em1330_v_dialogue.pck.3.x64.en

18217131131 18251704718
18220993113 em1370_v_dialogue.pck.3.x64.en

20180356416 20241899672
20187235043 em4103_v_dialogue.pck.3.x64.en

20079047866 20126111173
20079047866 em4105_v_dialogue.pck.3.x64.en

975582092 1008712930
975582092 pl2000_dialogue.pck.3.x64.en

3261758961 3291738439
3265076495 movie_bonus_dialog.pck.3.x64.en

18082008046 18103123619
18084365781 em4130_v_dialogue.pck.3.x64.en

18606474237 18632973839
18606474237 em4104_v_dialogue.pck.3.x64.en

18152135581 18171592518
18152135581 em4090_v_dialogue.pck.3.x64.en

18171592519 18216296126
18171592519 em4000_v_dialogue.pck.3.x64.en

20126111174 20144293480
20126111174 em4100_v_dialogue.pck.3.x64.en

20144293481 20162068318
20144293481 em4101_v_dialogue.pck.3.x64.en

20162068319 em4102_v_dialogue.pck.3.x64.en
20162068319 20180356415

18438588282 18454440445
18440377385 em4110_v_dialogue.pck.3.x64.en

6619741253 6630105302
6620935170 em4430_v_dialogue.pck.3.x64.en

6601679701 6612262879
6602864723 em4400_v_dialogue.pck.3.x64.en

956084055 967211189
956084055 em4122_v_dialogue.pck.3.x64.en

945198973 956084054
945198973 em4121_v_dialogue.pck.3.x64.en

937121945 em4120_v_dialogue.pck.3.x64.en
937121945 945198972

18380860642 18394264859
18382352586 em4070_v.pck.3.x64.en

5458135716 5506437540
5458135716 em1261_v_dialogue.pck.3.x64.en

[Masquerade]

Start size: 40.3GB

Main game files:

xcrilayla+xzlib+srep+lolz

The z files inside the cpks are compressed with zlib, read more here: https://forum.xentax.com/viewtopic.php?f=10&t=16011

Videos:

srep+lzma

Ofc you can recode them to save space, but then lossy

Audio:

msc+srep+lzma

These .wsp files are just multiple audio files concatenated. You can use this BMS script to "extract" them, however it's imperfect. To "pack" the files again, you can use bincat by aluigi or copy /b. Not much point in trying oggre_wwise, I got better result with msc

Final size: 20.2GB

[:( Sad8669]

panker1992

So if i want to rip an optional .wmv i have to use HeX to replace its values with 00 and then insert it back into the re_chunk_000.pak file?

I am afraid i am missing something big.

[Masquerade]

Originally Posted by :( Sad8669

panker1992

So if i want to rip an optional .wmv i have to use HeX to replace its values with 00 and then insert it back into the re_chunk_000.pak file?

I am afraid i am missing something big.

You are correct.

First, we start by extracting the pak file to see what's inside.

Next you find the video/file whatever you want to rip out.

Open HxD and drag into it your pak file - the "big file" - and your the file you want to rip.

Next we highlight the first few lines of the rip file on the hex side (do NOT select the decoded text side) and CTRL+C to copy the data to clipboard.

We move to the big file and CTRL+F and go to "Hex values" tab. We paste what we just copied and hit ENTER.

HxD will search the file for the matching hex values in the file. Depending on the size of the file + amount of data you selected, this can take quite long!

When HxD finds the data, it will show highlighted.

Next, we go to the top of the window and change where it says "hex" to "dec" (this is because SFK only supports decimal offsets).

Go to the start of the highlighted data that hxd presents you with, and you can right click and click Copy Offset (or ALT+INS on keyboard).

Open a notepad doc, nothing special, and paste the offset.

After copying the first offset, go back to the rip file and go right to the very bottom and copy from the bottom upwards some hex data (be careful, some videos such as binks can have repeating data at the end so you will need to copy until the data starts to differ).

Go to the big file and search again with this new hex values. Note that here, it's a good idea to set the search filter to forwards only since going backwards is useless and wastes our time!

Once you find the end offset, copy it again and paste it into the notepad.

I like to sort mine a little like this:

8476305309 - 8490311604 - example.mp4

7249864117 - 7615226644 - bonus.mp4

Now comes the patch out part. You right click in HxD in the big file and click Select Block (or CTRL+E).

Make sure your HxD is set to Dec and not hex otherwise this next step won't work (you will get an error saying "file does not contain offset")

In the first box, paste the first offset we copied. In the 2nd box, paste the second offset we copied. Ignore the length box, HxD determines it on it's own.

Now select the block and you now have the full ripped file selected within the big file.

Next, we right click, and click "Fill Selection". Choose to fill with 00s (this is selected by default, so you can just press ENTER in the window that appears). 00 is just empty space.

You will notice the hex data that was there previously now goes all to red 00s. The data has been removed. Now you can hit CTRL+S to save the file. HxD is very kind to you and makes a backup file with .bak extension of the unchanged file (again, can take a while depending on the size of the file). After done, the red 00s will become black. The file has saved. Congrats! You just ripped out a video from a big file.

To patch the video back in, we can use sfk - bat file:

sfk partcopy <ripped_file> -allfrom 0 <big_file> X -yes

Replace <ripped_file> and <big_file> with their names accordingly. Replace X with the first decimal offset we copied.

A completed bat would look something like this:

sfk partcopy example.mp4 -allfrom 0 mygame.pak 8476305309 -yes

-allfrom 0 tells SFK to copy data from the very start of the supplied file. -yes tells sfk to actually do the specified command, since otherwise it wouldn't do it.

[:( Sad8669]

Can't thank you enough for taking your time out for me and the community.
I know how painful it is to demonstrate a Practical thing Theoretically.

This whole para, i am gonna read it carefully when i get to it.

[L33THAK0R]

Originally Posted by Masquerade

You are correct.

First, we start by extracting the pak file to see what's inside.

Is there a public collection of file-lists for REEngine titles? Both the QuickBMS and RETool unpacking functions require a list to unpack the ".pak" archives and I can only seem to find such a list for "Devil May Cry 5".

[:( Sad8669]

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

[L33THAK0R]

Originally Posted by :( Sad8669

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

I've tried and succeeded in encoding ripped videos with block information provided by panker with RE2 Remake (I don't have access to a database/list for RE2s archives), and I'm fairly confident the REEngine iteration used by RE8 also allows for this. Personally I'd offer the end-user the choice between lossless & lossy videos, which is where patching the original files back in would be beneficial. I'm fairly certain your final compressor of choice would be able to take advantage of the massive amount of blank data within the archive to achieve a smaller output.

[Masquerade]

Originally Posted by :( Sad8669

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

You're confusing replacing the space in the big file with empty space with removing the space altogether.

Imagine you have 5 different coloured squares, each compress to a varying level of goodness:



Now, blue squares repesent videos, they don't compress well at all. So we use HxD to remove the blue square, leaving white empty space:



The big file size (represented by the black border around the squares) doesn't change. The space is still there, but nothing is occupying it, it's empty. Empty space compresses well, because there's nothing there to compress.

Since you took out a file that doesn't compress very well, you have just saved yourself loads of time when compressing under lolz, since lolz is slow. If you only use something like LZMA which is a bit faster, the speed gain becomes less prevalent.

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

For more of a general thing, yes.

Repacks intended to be lossless, but when you remove data from a file, you break CRC perfection. So, you put the files back in and everything is fine. Ofc you can keep it out if you want, you get a smaller repack in the end.

Some games have completely unused files inside the big files - take Just Cause 4 as example, instead of fixing their videos, they just re-rendered new videos and just slapped them into the patched data folders. The old videos are still in the game, but the game just doesn't read them. IIRC there's over 3GB of unused videos in that game (including the credits video to Just Cause 3!) - so ripping them out of the archives saves you not only time in compression and final compressed size.

Other times it's useful to just patch data out in order to compress it differently. E.g. you can patch the bink videos out of Just Cause 4 and compress them with bpk, whereas if you used lolz without tampering with the archives, you'd get a larger size since lolz doesn't work on bink videos.

Hope that explanation wasn't too all over the place

[Masquerade]

Originally Posted by L33THAK0R

Is there a public collection of file-lists for REEngine titles? Both the QuickBMS and RETool unpacking functions require a list to unpack the ".pak" archives and I can only seem to find such a list for "Devil May Cry 5".

https://residentevilmodding.boards.n...x-editing-tool

This may be useful to you

[L33THAK0R]

Originally Posted by Masquerade

https://residentevilmodding.boards.n...x-editing-tool

This may be useful to you

Thank you! This is quite the interesting list, I'll have to research how each entry is calculated (I'm guessing the process may be similar to the now abandoned, although fairly complete, "Mad Max" asset identification project that was started a while back, where each asset has to be called into memory and the resulting logs from Process Manager (windows) can be used to build a hash list), since it seems RE2 is only 2/3rds complete.