Best Compression Methods for 'Specific' Games. Q&A

[Masquerade]

Originally Posted by :( Sad8669

panker1992

So if i want to rip an optional .wmv i have to use HeX to replace its values with 00 and then insert it back into the re_chunk_000.pak file?

I am afraid i am missing something big.

You are correct.

First, we start by extracting the pak file to see what's inside.

Next you find the video/file whatever you want to rip out.

Open HxD and drag into it your pak file - the "big file" - and your the file you want to rip.

Next we highlight the first few lines of the rip file on the hex side (do NOT select the decoded text side) and CTRL+C to copy the data to clipboard.

We move to the big file and CTRL+F and go to "Hex values" tab. We paste what we just copied and hit ENTER.

HxD will search the file for the matching hex values in the file. Depending on the size of the file + amount of data you selected, this can take quite long!

When HxD finds the data, it will show highlighted.

Next, we go to the top of the window and change where it says "hex" to "dec" (this is because SFK only supports decimal offsets).

Go to the start of the highlighted data that hxd presents you with, and you can right click and click Copy Offset (or ALT+INS on keyboard).

Open a notepad doc, nothing special, and paste the offset.

After copying the first offset, go back to the rip file and go right to the very bottom and copy from the bottom upwards some hex data (be careful, some videos such as binks can have repeating data at the end so you will need to copy until the data starts to differ).

Go to the big file and search again with this new hex values. Note that here, it's a good idea to set the search filter to forwards only since going backwards is useless and wastes our time!

Once you find the end offset, copy it again and paste it into the notepad.

I like to sort mine a little like this:

8476305309 - 8490311604 - example.mp4

7249864117 - 7615226644 - bonus.mp4

Now comes the patch out part. You right click in HxD in the big file and click Select Block (or CTRL+E).

Make sure your HxD is set to Dec and not hex otherwise this next step won't work (you will get an error saying "file does not contain offset")

In the first box, paste the first offset we copied. In the 2nd box, paste the second offset we copied. Ignore the length box, HxD determines it on it's own.

Now select the block and you now have the full ripped file selected within the big file.

Next, we right click, and click "Fill Selection". Choose to fill with 00s (this is selected by default, so you can just press ENTER in the window that appears). 00 is just empty space.

You will notice the hex data that was there previously now goes all to red 00s. The data has been removed. Now you can hit CTRL+S to save the file. HxD is very kind to you and makes a backup file with .bak extension of the unchanged file (again, can take a while depending on the size of the file). After done, the red 00s will become black. The file has saved. Congrats! You just ripped out a video from a big file.

To patch the video back in, we can use sfk - bat file:

sfk partcopy <ripped_file> -allfrom 0 <big_file> X -yes

Replace <ripped_file> and <big_file> with their names accordingly. Replace X with the first decimal offset we copied.

A completed bat would look something like this:

sfk partcopy example.mp4 -allfrom 0 mygame.pak 8476305309 -yes

-allfrom 0 tells SFK to copy data from the very start of the supplied file. -yes tells sfk to actually do the specified command, since otherwise it wouldn't do it.

[L33THAK0R]

Originally Posted by Masquerade

You are correct.

First, we start by extracting the pak file to see what's inside.

Is there a public collection of file-lists for REEngine titles? Both the QuickBMS and RETool unpacking functions require a list to unpack the ".pak" archives and I can only seem to find such a list for "Devil May Cry 5".

[Masquerade]

Originally Posted by L33THAK0R

Is there a public collection of file-lists for REEngine titles? Both the QuickBMS and RETool unpacking functions require a list to unpack the ".pak" archives and I can only seem to find such a list for "Devil May Cry 5".

https://residentevilmodding.boards.n...x-editing-tool

This may be useful to you

[L33THAK0R]

Originally Posted by Masquerade

https://residentevilmodding.boards.n...x-editing-tool

This may be useful to you

Thank you! This is quite the interesting list, I'll have to research how each entry is calculated (I'm guessing the process may be similar to the now abandoned, although fairly complete, "Mad Max" asset identification project that was started a while back, where each asset has to be called into memory and the resulting logs from Process Manager (windows) can be used to build a hash list), since it seems RE2 is only 2/3rds complete.

[:( Sad8669]

Can't thank you enough for taking your time out for me and the community.
I know how painful it is to demonstrate a Practical thing Theoretically.

This whole para, i am gonna read it carefully when i get to it.

[:( Sad8669]

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

[L33THAK0R]

Originally Posted by :( Sad8669

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

I've tried and succeeded in encoding ripped videos with block information provided by panker with RE2 Remake (I don't have access to a database/list for RE2s archives), and I'm fairly confident the REEngine iteration used by RE8 also allows for this. Personally I'd offer the end-user the choice between lossless & lossy videos, which is where patching the original files back in would be beneficial. I'm fairly certain your final compressor of choice would be able to take advantage of the massive amount of blank data within the archive to achieve a smaller output.

[Masquerade]

Originally Posted by :( Sad8669

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

You're confusing replacing the space in the big file with empty space with removing the space altogether.

Imagine you have 5 different coloured squares, each compress to a varying level of goodness:



Now, blue squares repesent videos, they don't compress well at all. So we use HxD to remove the blue square, leaving white empty space:



The big file size (represented by the black border around the squares) doesn't change. The space is still there, but nothing is occupying it, it's empty. Empty space compresses well, because there's nothing there to compress.

Since you took out a file that doesn't compress very well, you have just saved yourself loads of time when compressing under lolz, since lolz is slow. If you only use something like LZMA which is a bit faster, the speed gain becomes less prevalent.

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

For more of a general thing, yes.

Repacks intended to be lossless, but when you remove data from a file, you break CRC perfection. So, you put the files back in and everything is fine. Ofc you can keep it out if you want, you get a smaller repack in the end.

Some games have completely unused files inside the big files - take Just Cause 4 as example, instead of fixing their videos, they just re-rendered new videos and just slapped them into the patched data folders. The old videos are still in the game, but the game just doesn't read them. IIRC there's over 3GB of unused videos in that game (including the credits video to Just Cause 3!) - so ripping them out of the archives saves you not only time in compression and final compressed size.

Other times it's useful to just patch data out in order to compress it differently. E.g. you can patch the bink videos out of Just Cause 4 and compress them with bpk, whereas if you used lolz without tampering with the archives, you'd get a larger size since lolz doesn't work on bink videos.

Hope that explanation wasn't too all over the place

[:( Sad8669]

Ahhh i see, that makes sense xD.

But as you said, when space is empty it compresses well, which means in the final compressed data, the size of the video will removed right?

and about the languages, panker1992 mentioned English Only and gave blocks of just ENG files but we are actually ripping languages other than English so how we get rid of other languages? or the ones mentioned were other than English?

Sorry for being a numbskull and Thanks!

[panker1992]

hello people

There is a serious business here and i have created a lil bit of chaos

BTW my Final Size for RE8 = 16.1GB


I am explaining and i will take RE8 For example

This game has total of 5.9 GB of audio plus videos files
It has 9 languages which as audible not subs of anything totalling 2.25 GB
Movies Total 1.79GB
Rest is ambient Music Game Just generic Music

Why in sevel hells would anyone with a mind compress this title altogether with lolz??

What I did was simple Remove all Videos and all Audios and ambient music etc and compressed textures aka tex files from the game down to 15 GB

then i saw what video files are optional and removed them around 1GB of videos are optional how the game is made and what artist and directors did etc so all of those gone

then i took the videos the game uses for the game and re encode them making them 70% smaller.

Then i took 8 out of 9 spoken languages and completely removed them that is another 2 GB in final size.

Then i said why not re encode the audio to a mid range wem files and save another 50% on the audio size.

you separate them all pack them and patch in the files at the end.

[L33THAK0R]

Originally Posted by panker1992

Then i said why not re encode the audio to a mid range wem files and save another 50% on the audio size.

Do you have a sample to listen to? I'm quite interested in how noticeable the quality loss is with ".wem" encoding!

[:( Sad8669]

This kind of interesting chaos is always welcome.

What i asked was that, are these audio files other than English?

Block 1
721910965 807697466
731727462 em1000_v_dialogue.pck.3.x64.en

Block 2
2422899086 2493439908 
2422899086 em1110_v_dialogue.pck.3.x64.en

Block 3
18114998188 18152135580
18119034210  em1150_v_dialogue.pck.3.x64.en

Block 4
20249490548 20294360582 
20254537491 em1230_v_dialogue.pck.3.x64.en

Block 5
19302022331 19349746976
19302022331 em1260_v_dialogue.pck.3.x64.en

Block 6
5506437541 5556561515
5506437541 em1262_v_dialogue.pck.3.x64.en

Block 7
841520222 871463738
841520222 em1300_v_dialogue.pck.3.x64.en

Block 8
20294360583 20341650467
20294360583 em1302_v_dialogue.pck.3.x64.en

Block 9
18394439764 em1310_v_dialogue.pck.3.x64.en
18394439764 18438588281

Block 10 -- this one is 500mb 
6100084475 6599324197
6100084475 pl1000_dialogue.pck.3.x64.en

Block 11 -- this one is also big 300mb 
156083303 470684395
191024214 movie_chapter3_dialog.pck.3.x64.en

Block 12  
20354417983 20417554742
20361475021 em1320_v_dialogue.pck.3.x64.en

Block 13  
18251704719 em4070_v_dialogue.pck.3.x64.en
18251704719 18380860641

Block 14  
18454440446 18491950561
18454440446 em1330_v_dialogue.pck.3.x64.en

Block 15  
18217131131 18251704718
18220993113 em1370_v_dialogue.pck.3.x64.en

Block 16 
20180356416 20241899672
20187235043 em4103_v_dialogue.pck.3.x64.en

Block 17
20079047866 20126111173
20079047866 em4105_v_dialogue.pck.3.x64.en

Block 18
975582092 1008712930
975582092 pl2000_dialogue.pck.3.x64.en

Block 19 Optional?
3261758961 3291738439
3265076495 movie_bonus_dialog.pck.3.x64.en

Block 20
18082008046 18103123619
18084365781 em4130_v_dialogue.pck.3.x64.en

Block 21
18606474237 18632973839
18606474237 em4104_v_dialogue.pck.3.x64.en

Block 22
18152135581 18171592518
18152135581 em4090_v_dialogue.pck.3.x64.en

Block 23
18171592519 18216296126
18171592519 em4000_v_dialogue.pck.3.x64.en 

Block 24
20126111174 20144293480 
20126111174 em4100_v_dialogue.pck.3.x64.en

Block 25
20144293481 20162068318
20144293481 em4101_v_dialogue.pck.3.x64.en

Block 26
20162068319 em4102_v_dialogue.pck.3.x64.en
20162068319 20180356415

Block 27
18438588282 18454440445
18440377385 em4110_v_dialogue.pck.3.x64.en

Block 28
6619741253 6630105302
6620935170 em4430_v_dialogue.pck.3.x64.en

Block 29
6601679701 6612262879
6602864723 em4400_v_dialogue.pck.3.x64.en

Block 30
956084055 967211189
956084055 em4122_v_dialogue.pck.3.x64.en

Block 31
945198973 956084054
945198973 em4121_v_dialogue.pck.3.x64.en

Block 32
937121945 em4120_v_dialogue.pck.3.x64.en
937121945 945198972

Block 33
18380860642 18394264859
18382352586 em4070_v.pck.3.x64.en

Block 34
5458135716 5506437540
5458135716 em1261_v_dialogue.pck.3.x64.en

and also why rip BGM? doesn't it make the game feel weird xD?

[panker1992]

Well,
The blocks Remove entire 9 languages at once and i chose to patch back only ENGLISH

i can show you how to patch another language instead of english if that is what you desire
GOD- i need a team chat so i can have a ncie talk with you guys

Leethakor sample is here
https://drive.google.com/file/d/1d86...ew?usp=sharing


If you find any difference i will give you my tricks and tools


PS Edit i forgot BGM = Background Music and it is patched back in after decompression it doesnt get lost

[L33THAK0R]

Originally Posted by panker1992

Leethakor sample is here
https://drive.google.com/file/d/1d86...ew?usp=sharing

Wow that's pretty incredible how little the loss in quality is to the naked human ear! I'll have to start experimenting with encoded audio now! Thankfully ".wem" files are fairly easy to encode (save for titles that use some older (~2008 - 2009) Vorbis codecs).

[:( Sad8669]

Which Blocks are the English ones (those you patch back) lmao.

This shit is getting crazier.

Also if i extract a file and make it 00 with hex, and then patch it back, will it be the same as doing the offset process?