Very Advanced Packing techniques - SFK Live Patch. 'Part 2'

[panker1992]

This is basically a Part 2 from my previous 'Very Advanced Way To Pack Part 1' (https://fileforums.com/showthread.php?t=103262) - repack data games files. with active patching SFK edition.

I will be using Resident Evil 2 Remaster, this methodology works on other games like Re 7, will work on RE 3, and even devil may cry 5.

Prons :

1) You can actually implement SFK with exec2 on your installer to live patch files.

2) You can Live patch whatever files you need with selective fitgirl style.

3) Its faster to Patch several hundreds of data into a gigantic 24GB file or in DMC 37GB i think instead of using hdiffz or xdelta which actually copies the data back and forth resulting in IO stress and delays.

4) you can earn a lot of time if you use LOLZ and other similar high density packing tools which take seriously LOOOONG time to pack.
Example = Re 2 24GB main has 3GB of movies and sounds which lolz can do nothing about.


Cons :


1) you need to use BMS to first extract data in order to search for them in the BIG file. if you go and use hex editor without firstly extracting Data, it will result into a mess you wont be able to restore and it may brick the headers on several large files.

2) it's time consuming and absolutely tiring to remove Data from a big file manually.

3) CRC will only be the same if you decide to actually keep all the data. and then patch them back into their position as they were.

SFK lacks a good documentation, you wont go very far without patience.

Step 1: Creating the Patch dataset files out of the main game.

Here is the command for creating the files out of the main game file.

sfk partcopy re_chunk_000.ok -fromto 991941787 1309235054 1.dat -yes
sfk partcopy re_chunk_000.ok -fromto 1387942086 1466133300 2.dat -yes
sfk partcopy re_chunk_000.ok -fromto 2629471608 2629561851 3.dat -yes
sfk partcopy re_chunk_000.ok -fromto 4919103905 4919192237 4.dat -yes
sfk partcopy re_chunk_000.ok -fromto 5918975763 5926516115 5.dat -yes
sfk partcopy re_chunk_000.ok -fromto 7467262974 8016575065 6.dat -yes
sfk partcopy re_chunk_000.ok -fromto 13882812163 13908248655 7.dat -yes
sfk partcopy re_chunk_000.ok -fromto 15436675783 15479560042 8.dat -yes
sfk partcopy re_chunk_000.ok -fromto 16568051778 16901655275 9.dat -yes
sfk partcopy re_chunk_000.ok -fromto 17643650733 17683414153 10.dat -yes
sfk partcopy re_chunk_000.ok -fromto 18313782043 18320168163 11.dat -yes
sfk partcopy re_chunk_000.ok -fromto 18540763045 18554523806 12.dat -yes
sfk partcopy re_chunk_000.ok -fromto 18799893563 18804743503 13.dat -yes
sfk partcopy re_chunk_000.ok -fromto 18938816495 19060992814 14.dat -yes
sfk partcopy re_chunk_000.ok -fromto 19243031386 19252085607 15.dat -yes
sfk partcopy re_chunk_000.ok -fromto 19353777849 19428629810 16.dat -yes
sfk partcopy re_chunk_000.ok -fromto 19546858625 20062292905 17.dat -yes
sfk partcopy re_chunk_000.ok -fromto 20091240616 20102471858 18.dat -yes
sfk partcopy re_chunk_000.ok -fromto 20711430589 20724952648 19.dat -yes
sfk partcopy re_chunk_000.ok -fromto 20977803823 21011127272 20.dat -yes
sfk partcopy re_chunk_000.ok -fromto 21228966153 21260371089 21.dat -yes
sfk partcopy re_chunk_000.ok -fromto 21345103962 21358949231 22.dat -yes
sfk partcopy re_chunk_000.ok -fromto 21924124909 21924470906 23.dat -yes
sfk partcopy re_chunk_000.ok -fromto 22312280532 22565588474 24.dat -yes
sfk partcopy re_chunk_000.ok -fromto 22770020588 22960399267 25.dat -yes
sfk partcopy re_chunk_000.ok -fromto 23008205405 23067241795 26.dat -yes
sfk partcopy re_chunk_000.ok -fromto 23722233838 23723711896 27.dat -yes
sfk partcopy re_chunk_000.ok -fromto 23723716037 23723741902 28.dat -yes

:: Optional unused videos


sfk partcopy re_chunk_000.ok -fromto 1387942084 1466133382 00.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 16901565029 17111549578 01.mp4 -yes


:: Real deal Videos


sfk partcopy re_chunk_000.ok -fromto 1473075626 1520109847 1.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 1520109847 1578513202 2.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 1578513202 1647569447 3.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 1647569447 1761563746 4.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 1761563746 1894174311 5.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 1894174311 1995046460 6.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 1995046460 2101462743 7.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 2101462743 2121790286 8.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 2121790286 2203430303 9.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 10134806181 10152118496 10.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 15437487147 15437692270 11.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 15437946371 15438143494 12.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 15890433073 15965574162 13.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 16080083460 16289822235 14.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 16435237595 16568051048 15.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 17111549578 17303329159 16.mp4 -yes
sfk partcopy re_chunk_000.ok -fromto 17303329159 17474394348 17.mp4 -yes

pause

Step 2: Patching Main game file with 00 Hex

You need to check on my other Post and see how to select a Block and remove the data.


Step 3: Use sfk with a bat or live patch the file using exec2 from your setup.


sfk.exe partcopy 1.mp4 -allfrom 0 re_chunk_000.pak 1473075626 -yes
sfk.exe partcopy 2.mp4 -allfrom 0 re_chunk_000.pak 1520109847 -yes
sfk.exe partcopy 3.mp4 -allfrom 0 re_chunk_000.pak 1578513202 -yes
sfk.exe partcopy 4.mp4 -allfrom 0 re_chunk_000.pak 1647569447 -yes
sfk.exe partcopy 5.mp4 -allfrom 0 re_chunk_000.pak 1761563746 -yes
sfk.exe partcopy 6.mp4 -allfrom 0 re_chunk_000.pak 1894174311 -yes
sfk.exe partcopy 7.mp4 -allfrom 0 re_chunk_000.pak 1995046460 -yes
sfk.exe partcopy 8.mp4 -allfrom 0 re_chunk_000.pak 2101462743 -yes
sfk.exe partcopy 9.mp4 -allfrom 0 re_chunk_000.pak 2121790286 -yes
sfk.exe partcopy 10.mp4 -allfrom 0 re_chunk_000.pak 10134806181 -yes
sfk.exe partcopy 11.mp4 -allfrom 0 re_chunk_000.pak 15437487147 -yes
sfk.exe partcopy 12.mp4 -allfrom 0 re_chunk_000.pak 15437946371 -yes
sfk.exe partcopy 13.mp4 -allfrom 0 re_chunk_000.pak 15890433073 -yes
sfk.exe partcopy 14.mp4 -allfrom 0 re_chunk_000.pak 16080083460 -yes
sfk.exe partcopy 15.mp4 -allfrom 0 re_chunk_000.pak 16435237595 -yes
sfk.exe partcopy 16.mp4 -allfrom 0 re_chunk_000.pak 17111549578 -yes
sfk.exe partcopy 17.mp4 -allfrom 0 re_chunk_000.pak 17303329159 -yes

sfk partcopy 1.dat -allfrom 0 re_chunk_000.pak 991941787 -yes
sfk partcopy 2.dat -allfrom 0 re_chunk_000.pak 1387942086 -yes
sfk partcopy 3.dat -allfrom 0 re_chunk_000.pak 2629471608 -yes
sfk partcopy 4.dat -allfrom 0 re_chunk_000.pak 4919103905 -yes
sfk partcopy 5.dat -allfrom 0 re_chunk_000.pak 5918975763 -yes
sfk partcopy 6.dat -allfrom 0 re_chunk_000.pak 7467262974 -yes
sfk partcopy 7.dat -allfrom 0 re_chunk_000.pak 13882812163 -yes
sfk partcopy 8.dat -allfrom 0 re_chunk_000.pak 15436675783 -yes
sfk partcopy 9.dat -allfrom 0 re_chunk_000.pak 16568051778 -yes
sfk partcopy 10.dat -allfrom 0 re_chunk_000.pak 17643650733 -yes
sfk partcopy 11.dat -allfrom 0 re_chunk_000.pak 18313782043 -yes
sfk partcopy 12.dat -allfrom 0 re_chunk_000.pak 18540763045 -yes
sfk partcopy 13.dat -allfrom 0 re_chunk_000.pak 18799893563 -yes
sfk partcopy 14.dat -allfrom 0 re_chunk_000.pak 18938816495 -yes
sfk partcopy 15.dat -allfrom 0 re_chunk_000.pak 19243031386 -yes
sfk partcopy 16.dat -allfrom 0 re_chunk_000.pak 19353777849 -yes
sfk partcopy 17.dat -allfrom 0 re_chunk_000.pak 19546858625 -yes
sfk partcopy 18.dat -allfrom 0 re_chunk_000.pak 20091240616 -yes
sfk partcopy 19.dat -allfrom 0 re_chunk_000.pak 20711430589 -yes
sfk partcopy 20.dat -allfrom 0 re_chunk_000.pak 20977803823 -yes
sfk partcopy 21.dat -allfrom 0 re_chunk_000.pak 21228966153 -yes
sfk partcopy 22.dat -allfrom 0 re_chunk_000.pak 21345103962 -yes
sfk partcopy 23.dat -allfrom 0 re_chunk_000.pak 21924124909 -yes
sfk partcopy 24.dat -allfrom 0 re_chunk_000.pak 22312280532 -yes
sfk partcopy 25.dat -allfrom 0 re_chunk_000.pak 22770020588 -yes
sfk partcopy 26.dat -allfrom 0 re_chunk_000.pak 23008205405 -yes
sfk partcopy 27.dat -allfrom 0 re_chunk_000.pak 23722233838 -yes
sfk partcopy 28.dat -allfrom 0 re_chunk_000.pak 23723716037 -yes

sfk partcopy "920567\re_dlc_000.dat" -allfrom 0 "920567\re_dlc_000.pak" 592 -yes


Exec2 Paradigm

if not ISExec ( 0, 0, 0, ExpandConstant('{tmp}\sfk.exe'), ExpandConstant(' partcopy 28.dat -allfrom 0 re_chunk_000.pak 23723716037 -yes'), ExpandConstant('{app}'), 'Rebuilding...Please_Wait',false) then break;

[L33THAK0R]

This is basically a Part 2 from my previous 'Very Advanced Way To Pack Part 1' (https://fileforums.com/showthread.php?t=103262) - repack data games files. with active patching SFK edition.

I will be using Resident Evil 2 Remaster, this methodology works on other games like Re 7, will work on RE 3, and even devil may cry 5.

Could I ask how you were able to find/generate the pak names required to unpack the archives for this title? From the little research I've conducted it seems a file list is required for unpacking (Using either the RE-Engine QuickBMS unpack script or RETool), but I can't seem to find any list with the addresses for any RE-Engine title's video files.

[Dragonis40]

Good morning! Is there any guide to use this tool?
If I have a generic videogame with generic .xxx files, how can i use this tool on videogame?
Starting from an installed videogame, which are the needed steps to compress videogame using SFK?
What does "sfk partcopy re_chunk_000.ok -fromto 991941787 1309235054 1.dat -yes" and all other instructions mean?
Unfortunately, these tools are almost never clear :(

[L33THAK0R]

Good morning! Is there any guide to use this tool?
If I have a generic videogame with generic .xxx files, how can i use this tool on videogame?
Starting from an installed videogame, which are the needed steps to compress videogame using SFK?
What does "sfk partcopy re_chunk_000.ok -fromto 991941787 1309235054 1.dat -yes" and all other instructions mean?
Unfortunately, these tools are almost never clear :(

In layman's terms this command is copying all the data from the offset (in decimal) 991941787 to the offset 1309235054, and saving the output into a new file named 1.dat

This tool is only useful if you're trying to rip specific chunks of data to either re-encode or to pack separately. An example of the latter would be if you find say some audio data or a Bink video file that can have their applicable compressor applied. The reason we do this is simply adding an additional and specific compressor to an existing chain can introduce instability and possible corruption as well as potentially resulting in a worse final compression ratio.

From the brief scenario you've outlined I don't believe your project would benefit from this method, however if you provide more detail I could very much be wrong.

[wareck]

Hello
I am studying this method...
indeed i get good results on block processing (ogg, png)
i've also realized that softs like oggre or lolz compress a large block better than several small files
(for example, for oggre, deduplication is better on a large file than on individual small files)

in short, I like it, it's time-consuming to build and takes a bit longer to decompress, but it works well.

I'll take this opportunity to share the little program I compiled from source code this weekend.

It's sfk's partcopy function alone.
It just saves a few MB on the final archive by not copying sfk in its entirety.



to use it :

partcopy re_chunk_000.ok -fromto 991941787 1309235054 1.dat


(remove sfk at begin and -yes at the end...


@For L33THAK0R:
this instruction is for copying a part of a big file (re_chunk), we take a portion begin at 991941787 and finish at 1309235054 and export this bloc to a new file called 1.dat

in simple words, we concatenate small files in a big one , compress with your favorite program, because you concatenate same kind of file, optimized compressor will give better results

then during install decompress then separate files

best regards