NieR:Automata™ [Game of the YoRHa Edition] [Build 7020666]
 

Start size: 40.3GB

Main game files:

The z files inside the cpks are compressed with zlib, read more here: https://forum.xentax.com/viewtopic.php?f=10&t=16011

Videos:

Ofc you can recode them to save space, but then lossy :)

Audio:

These .wsp files are just multiple audio files concatenated. You can use this BMS script to "extract" them, however it's imperfect. To "pack" the files again, you can use bincat by aluigi or copy /b. Not much point in trying oggre_wwise, I got better result with msc

Final size: 20.2GB

panker1992

So if i want to rip an optional .wmv i have to use HeX to replace its values with 00 and then insert it back into the re_chunk_000.pak file?

I am afraid i am missing something big.

You are correct.

First, we start by extracting the pak file to see what's inside.

Next you find the video/file whatever you want to rip out.

Open HxD and drag into it your pak file - the "big file" - and your the file you want to rip.

Next we highlight the first few lines of the rip file on the hex side (do NOT select the decoded text side) and CTRL+C to copy the data to clipboard.

We move to the big file and CTRL+F and go to "Hex values" tab. We paste what we just copied and hit ENTER.

HxD will search the file for the matching hex values in the file. Depending on the size of the file + amount of data you selected, this can take quite long!

When HxD finds the data, it will show highlighted.

Next, we go to the top of the window and change where it says "hex" to "dec" (this is because SFK only supports decimal offsets).

Go to the start of the highlighted data that hxd presents you with, and you can right click and click Copy Offset (or ALT+INS on keyboard).

Open a notepad doc, nothing special, and paste the offset.

After copying the first offset, go back to the rip file and go right to the very bottom and copy from the bottom upwards some hex data (be careful, some videos such as binks can have repeating data at the end so you will need to copy until the data starts to differ).

Go to the big file and search again with this new hex values. Note that here, it's a good idea to set the search filter to forwards only since going backwards is useless and wastes our time!

Once you find the end offset, copy it again and paste it into the notepad.

I like to sort mine a little like this:

Now comes the patch out part. You right click in HxD in the big file and click Select Block (or CTRL+E).

Make sure your HxD is set to Dec and not hex otherwise this next step won't work (you will get an error saying "file does not contain offset")

In the first box, paste the first offset we copied. In the 2nd box, paste the second offset we copied. Ignore the length box, HxD determines it on it's own.

Now select the block and you now have the full ripped file selected within the big file.

Next, we right click, and click "Fill Selection". Choose to fill with 00s (this is selected by default, so you can just press ENTER in the window that appears). 00 is just empty space.

You will notice the hex data that was there previously now goes all to red 00s. The data has been removed. Now you can hit CTRL+S to save the file. HxD is very kind to you and makes a backup file with .bak extension of the unchanged file (again, can take a while depending on the size of the file). After done, the red 00s will become black. The file has saved. Congrats! You just ripped out a video from a big file.

To patch the video back in, we can use sfk - bat file:

Replace <ripped_file> and <big_file> with their names accordingly. Replace X with the first decimal offset we copied.

A completed bat would look something like this:

-allfrom 0 tells SFK to copy data from the very start of the supplied file. -yes tells sfk to actually do the specified command, since otherwise it wouldn't do it.

Can't thank you enough for taking your time out for me and the community.
I know how painful it is to demonstrate a Practical thing Theoretically.

This whole para, i am gonna read it carefully when i get to it.

Is there a public collection of file-lists for REEngine titles? Both the QuickBMS and RETool unpacking functions require a list to unpack the ".pak" archives and I can only seem to find such a list for "Devil May Cry 5".

Ok so i read the whole process, everything went smoothly.

Lets talk about RE8 specifically. Ripping the video outta the big file using HxD doesn't reduce its actual size.

Then what is the point?

Or does it decreases upon compressing?

Also why patch files back once ripped? they are optional anyway, or was that a reference for universal use?

I've tried and succeeded in encoding ripped videos with block information provided by panker with RE2 Remake (I don't have access to a database/list for RE2s archives), and I'm fairly confident the REEngine iteration used by RE8 also allows for this. Personally I'd offer the end-user the choice between lossless & lossy videos, which is where patching the original files back in would be beneficial. I'm fairly certain your final compressor of choice would be able to take advantage of the massive amount of blank data within the archive to achieve a smaller output.

You're confusing replacing the space in the big file with empty space with removing the space altogether.

Imagine you have 5 different coloured squares, each compress to a varying level of goodness:

https://i.ibb.co/RjFhVnx/Untitled.png

Now, blue squares repesent videos, they don't compress well at all. So we use HxD to remove the blue square, leaving white empty space:

https://i.ibb.co/Jmkjpwd/Untitled.png

The big file size (represented by the black border around the squares) doesn't change. The space is still there, but nothing is occupying it, it's empty. Empty space compresses well, because there's nothing there to compress.

Since you took out a file that doesn't compress very well, you have just saved yourself loads of time when compressing under lolz, since lolz is slow. If you only use something like LZMA which is a bit faster, the speed gain becomes less prevalent.

For more of a general thing, yes.

Repacks intended to be lossless, but when you remove data from a file, you break CRC perfection. So, you put the files back in and everything is fine. Ofc you can keep it out if you want, you get a smaller repack in the end.

Some games have completely unused files inside the big files - take Just Cause 4 as example, instead of fixing their videos, they just re-rendered new videos and just slapped them into the patched data folders. The old videos are still in the game, but the game just doesn't read them. IIRC there's over 3GB of unused videos in that game (including the credits video to Just Cause 3!) - so ripping them out of the archives saves you not only time in compression and final compressed size.

Other times it's useful to just patch data out in order to compress it differently. E.g. you can patch the bink videos out of Just Cause 4 and compress them with bpk, whereas if you used lolz without tampering with the archives, you'd get a larger size since lolz doesn't work on bink videos.

Hope that explanation wasn't too all over the place :D

https://residentevilmodding.boards.n...x-editing-tool

This may be useful to you

Thank you! This is quite the interesting list, I'll have to research how each entry is calculated (I'm guessing the process may be similar to the now abandoned, although fairly complete, "Mad Max" asset identification project that was started a while back, where each asset has to be called into memory and the resulting logs from Process Manager (windows) can be used to build a hash list), since it seems RE2 is only 2/3rds complete.

Ahhh i see, that makes sense xD.

But as you said, when space is empty it compresses well, which means in the final compressed data, the size of the video will removed right?

and about the languages, panker1992 mentioned English Only and gave blocks of just ENG files but we are actually ripping languages other than English so how we get rid of other languages? or the ones mentioned were other than English?

Sorry for being a numbskull and Thanks!

hello people :D

There is a serious business here and i have created a lil bit of chaos :D

BTW my Final Size for RE8 = 16.1GB


I am explaining :D and i will take RE8 For example

This game has total of 5.9 GB of audio plus videos files
It has 9 languages which as audible not subs of anything totalling 2.25 GB
Movies Total 1.79GB
Rest is ambient Music Game Just generic Music

Why in sevel hells would anyone with a mind compress this title altogether with lolz??

What I did was simple Remove all Videos and all Audios and ambient music etc and compressed textures aka tex files from the game down to 15 GB

then i saw what video files are optional and removed them around 1GB of videos are optional how the game is made and what artist and directors did etc so all of those gone

then i took the videos the game uses for the game and re encode them making them 70% smaller.

Then i took 8 out of 9 spoken languages and completely removed them that is another 2 GB in final size.

Then i said why not re encode the audio to a mid range wem files and save another 50% on the audio size.

you separate them all pack them and patch in the files at the end.

This kind of interesting chaos is always welcome.

What i asked was that, are these audio files other than English?

and also why rip BGM? doesn't it make the game feel weird xD?

Do you have a sample to listen to? I'm quite interested in how noticeable the quality loss is with ".wem" encoding!

Well,
The blocks Remove entire 9 languages at once and i chose to patch back only ENGLISH

i can show you how to patch another language instead of english if that is what you desire
GOD- i need a team chat so i can have a ncie talk with you guys

Leethakor sample is here
https://drive.google.com/file/d/1d86...ew?usp=sharing


If you find any difference i will give you my tricks and tools :D


PS Edit i forgot :D BGM = Background Music and it is patched back in after decompression :D it doesnt get lost

Which Blocks are the English ones (those you patch back) lmao.

This shit is getting crazier.

Also if i extract a file and make it 00 with hex, and then patch it back, will it be the same as doing the offset process?

What you need isnt a block for language

Block = the area to cut with hex from and to from point A to point B offset 01 to offset 500 example given.

What you need to patch a file is the file to patch and the starting offset where it starts inside the gigantic file!!

it's practical usage have you seen my youtube videos where i explain how this works ?


You can pack a game Fitgirl Size if you master what i am showing here

Please refer me toward the video.

Check Messages and you will find it

Wow that's pretty incredible how little the loss in quality is to the naked human ear! I'll have to start experimenting with encoded audio now! Thankfully ".wem" files are fairly easy to encode (save for titles that use some older (~2008 - 2009) Vorbis codecs).

what you saw is high quality 400kbps all the way to low quality 128kbps

i save many GB by doing this :P

I look forward to being able to share some similarly impressive results in the near future!

Steel Division 2 [Build 7022650]
 

Start Size: 60.4GB

For bonus content:

Final Size: 29.2GB (29.4GB with Bonus Content)

This took 18-19 hours for me in total.

UPDATE: It seems it was an error on my end after-all.

Has anyone else had issues with reflate & preflate regarding "The Saboteur"? I ran reflate for a good 14hrs prior, as a first test, but alas it was still stuck at the same spot. Just want to know if it's something on my end or just the title itself, as the "Odin Engine" is a custom game-engine only used for this title (I don't know if its a fork of something like UE3 or Source, I couldn't find any documentation on it, after a brief searching period), and I'm not sure how its game-data archives function.

Try using old xtool. Also use stdio instead of using $$arc$$s, it's faster. Also, srep+lolz+bpk makes bpk useless since any bink data will be put through srep and lolz.

Someone knows a good compressor for gif images? If possible.

Yeah ended up going with xtool 2019.

I'm still a massive newbie so got no clue what stdio is, I'll have to research this

I'm fully aware, the actual order of compressors for that preset is bpk + srep + lolz, but as its my own script I haven't seen the need to change that (I have ~40 copies of it on my system, spread across multiple drives to limit the impact of storage I/O limitations, making it a pain to update all of em since at least 7 instances are running at any given time). Now that I think about it I used to use a mask for the bpk compressor (srep+lolz/$BINK=bpk) until a few months back when I encountered bink video files under different extension names & within archives, and stopped using it as a mask (couldn't be bothered constantly updating the mask), which probably explains the label of the preset.

flexiGIF - Lossless GIF/LZW optimization
 

You maybe want to take a look at this:
flexiGIF - Lossless GIF/LZW optimization

Very slow won't describe on how slow it actually is, not at all. Holy sh...

Precomp supports gifs, run precomp+srep+your choiceof compress

Instead of $$arcdatafile$$.tmp $$arcpackedfile$$.tmp,you use - - <stdin> <stdout>

Example for old xtool:

Ah alright, I didn't think there was an actual performance difference between the two, I'll run some tests to see what the difference is!

packcmd = xtool e:precomp:t100p,c128m:zlib - <stdin> $$arcpackedfile$$.tmp

according to my endless debug with Zee this one is the best way to avoid some problems

the file will have to be copied only once before srep actually takes it in. but its best in terms of debuggery

Muck
 

The Sims 4 [v1.77.131.1030 + All DLCs]
 

Start size: 47.7GB

xzlib+srep+lolz

Soundtrack: mpz

Compressed size: 29.2GB

Takes around ~15 hours.

x86 ripped.

xzlib+srep+4x4:lzma
30.4 GB

Takes around ~2 hours

PS: Ripped x86 stuff (Data_LE, Game_LE)

https://i.ibb.co/Yk9gM3s/I8Iay5H.jpg
+13 hours for -1.2GB :p

The pain.....

Although, two methods are appreciated in the INDEX :P.

Not a good time/output ratio if you ask me :D But I'm not a repacker who share his stuff :p

Therefore, for me 4x4:lzma is the best in time/output ratio in most cases. I just use lolz for testing stuff anyway.

Is there a way to compress .awb audio files?

1:1 CRC Perfect.