FileForums

FileForums (https://fileforums.com/index.php)
-   Conversion Tutorials (https://fileforums.com/forumdisplay.php?f=55)
-   -   Best Compression Methods for 'Specific' Games. Q&A (https://fileforums.com/showthread.php?t=99554)

Masquerade 13-04-2020 05:03

Saints Row Gat Out Of Hell
 
Total unpacked size: 5.52GB

Compression as follows:
  • *.vpp_pc =
    Code:

    srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba4096
  • *.bik =
    Code:

    bpk
  • *.bk2 =
    Code:

    srep
  • Everything else:
    Code:

    srep:m3f:l512+razor
Total packed size using above methods: 3.54GB

Notes: I couldn't get BPK to play nice (always general compression errors) with the *.bk2 video files, so I just used SREP. It's possible you could end up with a much smaller conversion if you can get BPK to work on the *.bk2 files.

KaktoR 13-04-2020 07:12

Could you send me a sample of this bk2 file?

Masquerade 13-04-2020 12:51

Quote:

Originally Posted by KaktoR (Post 485276)
Could you send me a sample of this bk2 file?

I messaged you but I'm not sure how I can see if it sent or not so here's the link just in case.

https://letsupload.co/3yy0f/BK2.7z - 300mb

There's varying sizes of files in there so you can get an idea of what came with the game. In total, there's 1.7GB of bk2 video.

Masquerade 14-04-2020 04:42

Fallout New Vegas Ultimate Edition
 
Total unpacked size: 9.22GB

Compression as follows:
  • *.bsa =
    Code:

    pZlib3:e+srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba4096
  • *.bik =
    Code:

    bpk
  • *.wav =
    Code:

    msc:wav=1:tak=9+srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba409
  • *.mp3 =
    Code:

    msc:wav=1:tak=9+srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba409
  • Everything else:
    Code:

    srep:m3f:l512+razor

Total packed size using above methods: 5.41GB

shazzla 14-04-2020 04:55

@Masquerade :
Srep+razor : whats the point ? Razor already dedups the data ! Use LOLZ instead of RZ. It compresses much better.

Mp3 : use MPZAPI with PMT. Much faster than MSC.
Use MSC when MP3 files are embedded in other files(stream).

BK2 : some bk2 files cannot be processed as bink with BPK (corrupts the archive). Compress them separately to check things.
Hope this was helpful. :)

KaktoR 14-04-2020 05:20

Quote:

Originally Posted by Masquerade (Post 485279)
I messaged you but I'm not sure how I can see if it sent or not so here's the link just in case.

https://letsupload.co/3yy0f/BK2.7z - 300mb

There's varying sizes of files in there so you can get an idea of what came with the game. In total, there's 1.7GB of bk2 video.

Thanks.

I was able to pack them into a archive using binkpack, but the problem is that the ratio is always 100%. I looked through file headers and they are fine. So maybe some special codec or something, if it's even possible with binkvideo.

Masquerade 14-04-2020 06:47

Quote:

Originally Posted by KaktoR (Post 485287)
Thanks.

I was able to pack them into a archive using binkpack, but the problem is that the ratio is always 100%. I looked through file headers and they are fine. So maybe some special codec or something, if it's even possible with binkvideo.

Good to know I'm not going insane (yet)!

It's strange, because it's the only Saints Row game in the series that uses Bink2 instead of Bink. Maybe that's just because Bink2 wasn't around when the other games were made, but then why not make all of the video Bink2? There's still old Bink files in the game (2 .bik files).

Masquerade 14-04-2020 06:50

Quote:

Originally Posted by shazzla (Post 485286)
@Masquerade :
Srep+razor : whats the point ? Razor already dedups the data ! Use LOLZ instead of RZ. It compresses much better.

Mp3 : use MPZAPI with PMT. Much faster than MSC.
Use MSC when MP3 files are embedded in other files(stream).

BK2 : some bk2 files cannot be processed as bink with BPK (corrupts the archive). Compress them separately to check things.
Hope this was helpful. :)

Thank you for the points! I have found that srep+razor yields better results with misc small files and binaries over LOLZ. I don't know if this is true for all games, but I will start to test both srep+rz against srep+lolz.

I'll look into PMT, will it have a major effect on conversion size?

Grumpy 14-04-2020 17:00

Quote:

If you're reading this, I meant to reply to shazzla and KatoR in one post, not 2! I am very sorry!
I dont mind 2 posts if they each add different info or if they are replies to different people ;)

shazzla 14-04-2020 23:34

@Masquerade :
AFAIK : RZ have some EXE/audio/etc optimized code. LOLZ optimized for DDS. For example...

KaktoR 15-04-2020 01:56

Quote:

Originally Posted by Masquerade (Post 485290)
I'll look into PMT, will it have a major effect on conversion size?

No, at least not for razor.

MPZ is minimal -> https://fileforums.com/showpost.php?...5&postcount=56

shazzla 15-04-2020 06:48

@KaktoR :

Not exactly....
With a really large dataset (maybe Quantum Break? I dont remember) ,the difference was soooo big :
PMT + RZ d128m / RZ d1023m.... 10 gigs or somethink like that...
So i stick with single thread !

JustFun 15-04-2020 07:01

1 Attachment(s)
Neighbors From Hell 1

Game Size: 122 MB

Compressed Game Size: 68.4 MB

Version: v1.0

Ripped: Nothing

Methods:

Code:

msc+srep on Data\(sfxdata.bnd and sfxdatahigh.bnd)

pzlib+srep+lolz on everything else

arc.ini attached

KaktoR 15-04-2020 07:30

Quote:

Originally Posted by shazzla (Post 485308)
@KaktoR :

Not exactly....
With a really large dataset (maybe Quantum Break? I dont remember) ,the difference was soooo big :
PMT + RZ d128m / RZ d1023m.... 10 gigs or somethink like that...
So i stick with single thread !

Are you sure?

I will test soon.

JustFun 15-04-2020 07:34

1 Attachment(s)
Neighbors From Hell 2

Game Size: 240 MB

Compressed Game Size: 132 MB

Version: v1.01

Ripped: Nothing

Methods:

Code:

msc+srep on Data\(sfxdata.bnd and sfxdatahigh.bnd)

pzlib+srep+lolz on everything else

arc.ini attached

turpro 15-04-2020 11:39

Code:

Half-Life 2 3in1(HL2&EP1&EP2)

Ripped: Nothing
Original Size:  8,05 GB
Compressed Size: 2,85 GB

Method:          msc:wav=1:raw=1:mp3=1:tak=9+srep:m5f:a2+lolz:d64m:mtt1:mt4:mc1023
                                                 
Compress Time:  ~50 minutes
Install Time:    3 minutes

CRC Checked:    Yes

Code:

Half-Life 2 Lost Coast

Ripped: Nothing
Original Size:  2,78 GB
Compressed Size: 1,01 GB

Method:          msc:wav=1:raw=1:mp3=1:tak=9+srep:m5f:a2+lolz:d64m:mtt1:mt4:mc1023
                                                 
Compress Time:  ~30 minutes
Install Time:    2 minutes

CRC Checked:    Yes

Code:

Half-Life WON
includes: Counter-Strike 1.5, Deathmatch Classic, Ricochet, Team Fortress Classic

Ripped: Nothing
Original Size:  781 MB
Compressed Size: 256 MB

Method:          srep:m5f:a1+lolz:d96m:mtt1:mt8:mc2
                                                 
Compress Time:  15 minutes
Install Time:    2 minutes

CRC Checked:    Yes


st0rm 16-04-2020 06:31

Dead Block [2011]
 
Code:

Game Version : 1.0

Ripped - Re-encoded Files : Nothing (Multi-5)

Original Size : 751 MB

Compressed Size : 231 MB

Install Time : 5 Mins (Depends your system)

Method : uelr:uv+srep:m3f+lzma


shazzla 18-04-2020 00:56

@KaktoR :

What i wrote before : XCom2 - WOTC ,*.upk+*.tfc. Preprocessed and repacked using RZ.

RZ 1023m single thread : 95.4 GB/12.7 GB
RZ PMT 4 threads/128m : 95.4 GB/26.5 GB

IMHO its because dedup/chunking.

KaktoR 18-04-2020 03:37

Thanks.

How long took it to compress without PMT?

shazzla 18-04-2020 04:47

Compression speed depends on the data of course. I dont know. Maybe 1.5 day on an I5-6600.

doofoo24 18-04-2020 22:23

Quote:

Originally Posted by shazzla (Post 485369)
Compression speed depends on the data of course...

you mean precompressor :confused:
Compression speed depends of course on algorithm (lzma2-zstd...) and hardware (cpu threads-ram speed-ssd)

shazzla 19-04-2020 03:00

No. I mean compression speed. In my experiences LOLZ pack many same bytes much faster than different bytes.

Titeuf 19-04-2020 05:27

Good morning all,
For the drug dealer simulator game, what compression method can you advise me, please?
Thank you in advance,
cordially

Masquerade 19-04-2020 05:48

Quote:

Originally Posted by Titeuf (Post 485381)
Good morning all,
For the drug dealer simulator game, what compression method can you advise me, please?
Thank you in advance,
cordially

It's built in unreal engine, so for the Pak file use xtool with zlib codec+srep+lolz then for everything else use srep+lolz.

So:

*.pak =

Code:

xtool:e:precomp:zlib+srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba4096
Everything else:

Code:

srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba4096
Enjoy the conversion!

Masquerade 19-04-2020 06:46

Stranger Things 3: The Game
 
Total unpacked size: 814mb

Compression as follows:

Code:

srep:m3f:l512+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba4096
Total packed size using above methods: 65mb

BobRogers21 23-04-2020 10:42

Which tool should I use?
 
Hello there, I am currently looking to compress the game "Astroneer". I just wish to see how small I can compress the game.

It is an Unreal Engine game, so I used Masquerade's guidance from the last page for Drug Dealer Simulator however I am wondering if there's any better compression method for mp4 video files? It seems that srep+lolz do not compress these files very well.

This forum has helped me a lot since my first post. Thank you in advance.

devil777 23-04-2020 13:13

It's better to Let alone those.... mp4 file.... just put em with whatever files your compressing with srep+lolz

Also you can reduce them in size by degrading their quality.... Like degrading to 720p from 1080p or higher .And you don't want that right?...

till now I have found no other tool to integrate with mp4 files without degrading their quality

ShiyaVivala 23-04-2020 17:53

1 Attachment(s)
Quote:

Originally Posted by BobRogers21 (Post 485429)
Hello there, I am currently looking to compress the game "Astroneer". I just wish to see how small I can compress the game.

It is an Unreal Engine game, so I used Masquerade's guidance from the last page for Drug Dealer Simulator however I am wondering if there's any better compression method for mp4 video files? It seems that srep+lolz do not compress these files very well.

This forum has helped me a lot since my first post. Thank you in advance.

You can use "uelr" to tfc files for precompressing.

arc.ini file:
Code:

[External compressor:uelr]
header    = 0
default    = -uv
packcmd    = "uelr_x86" {option} $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd  = "uelr_x86" r <stdin> <stdout>

Then srep+lolz.

For mp4 files you can create a batch script and re-encode them with FFmpeg.
https://github.com/FFmpeg/FFmpeg

ShiyaVivala 23-04-2020 18:03

Code:

Title:          Rome: Total War - Collection


Ripped Files:    Nothing

Original Size:  3,61 GB
Compressed Size: 1,44 GB

Method:          srep+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1:mtt1:mt4:d64m:fba4096

Install Time:    2 Min


Prettyboy099 23-04-2020 21:17

MotoGP 20
 
Input size - 18.7 GB

1. Method - srep+4x4:lzma
Compression time - 1.5 hours
Final size - 15.6 GB
Installation time - 3 min. 46 sec.

2. Method - xzlib+srep+lzma
Compression time - 2 hours 53 min.
Final size - 10.8 GB
Installation time - 13 min. 55 sec.

Masquerade 24-04-2020 06:31

The Amazing Spider Man 2 (2014) has a file called Streams2.dat, it's the largest file in the entire game (2,323,030,372 bytes / 2.16GB) and I don't know how to compress it efficiently.

I threw it into RAZOR12911's scanner and this was the result:

https://i.imgur.com/x8XrhoX.png

There's zlib streams so I decided to use XTool Zlib, but that didn't do anything:

Code:

Compressing 2 files, 2,323,030,372 bytes
  Compressing Data\
  Compressing Data\Streams2.dat
Compressed 2 files, 2,323,030,372 => 2,323,031,492 bytes. Ratio 100.00%
Compression time: cpu 2.23 sec/real 72.99 sec = 3%. Speed 31.83 mB/s
All OK

ZTool zlib was the same:

Code:

Compressing 2 files, 2,323,030,372 bytes
  Compressing Data\
  Compressing Data\Streams2.dat
Compressed 2 files, 2,323,030,372 => 2,323,035,496 bytes. Ratio 100.00%
Compression time: cpu 2.09 sec/real 50.60 sec = 4%. Speed 45.91 mB/s
All OK

PZLib also had a simlar result at a 100% ratio.

Is there something I'm missing here? There's also lots of WAV streams detected, but I don't know how to focus on them. I tried MSC, but that only found WAV streams of an unknown file type, and did nothing.

I don't think the file could be encrypted because the streams were detected.

If anyone has any tips, they'd be greatly appreciated.

Entai44 24-04-2020 15:16

XTool + MSC?

dixen 25-04-2020 04:26

Quote:

Originally Posted by Masquerade (Post 485446)
Is there something I'm missing here?

chunk size 128 or 256...try it

KaktoR 27-04-2020 02:52

FINAL FANTASY XII THE ZODIAC AGE
v1.4.0 - only 1 language

Code:

xZLib+srep+4x4:lzma

 2:14:03 - Unpacking file...
 2:47:24 - Unpacking file finished
 2:47:24 - Delete languages
 2:47:24 - Packing file
 3:49:49 - Packing file finished
 3:50:26 - Starting Data1.bin.001 creation
 5:03:18 - Finished Data1.bin.001 creation

Input size:  25.203g
Output size: 19.053g


Masquerade 28-04-2020 08:18

Quote:

Originally Posted by dixen (Post 485469)
chunk size 128 or 256...try it

I tried adding the c128 command into XTool and it slowed it right down, taking ages to go barely 0.1%.

devil777 28-04-2020 11:11

@Masaqurde... use "srep+lolz" method it'll worth the time I hope...

Also have you tried the x86 version?

Masquerade 28-04-2020 11:40

Quote:

Originally Posted by devil777 (Post 485510)
@Masaqurde... use "srep+lolz" method it'll worth the time I hope...

Also have you tried the x86 version?

I always use x86 version of XTool. I've never used x64 version. I just used srep and lolz and it came out at 1.9GB which is good enough for me. Very hard file to process.

Thank you all for the help.

KaktoR 29-04-2020 06:10

Quote:

Originally Posted by Masquerade (Post 485446)
The Amazing Spider Man 2 (2014) has a file called Streams2.dat, it's the largest file in the entire game (2,323,030,372 bytes / 2.16GB) and I don't know how to compress it efficiently.

I threw it into RAZOR12911's scanner and this was the result:

https://i.imgur.com/x8XrhoX.png

There's zlib streams so I decided to use XTool Zlib, but that didn't do anything:

Code:

Compressing 2 files, 2,323,030,372 bytes
  Compressing Data\
  Compressing Data\Streams2.dat
Compressed 2 files, 2,323,030,372 => 2,323,031,492 bytes. Ratio 100.00%
Compression time: cpu 2.23 sec/real 72.99 sec = 3%. Speed 31.83 mB/s
All OK

ZTool zlib was the same:

Code:

Compressing 2 files, 2,323,030,372 bytes
  Compressing Data\
  Compressing Data\Streams2.dat
Compressed 2 files, 2,323,030,372 => 2,323,035,496 bytes. Ratio 100.00%
Compression time: cpu 2.09 sec/real 50.60 sec = 4%. Speed 45.91 mB/s
All OK

PZLib also had a simlar result at a 100% ratio.

Is there something I'm missing here? There's also lots of WAV streams detected, but I don't know how to focus on them. I tried MSC, but that only found WAV streams of an unknown file type, and did nothing.

I don't think the file could be encrypted because the streams were detected.

If anyone has any tips, they'd be greatly appreciated.

What version of GFS is this? Because mine looks different.

Masquerade 29-04-2020 10:32

Quote:

Originally Posted by KaktoR (Post 485520)
What version of GFS is this? Because mine looks different.

https://i.imgur.com/GED9Fxh.png

I completely have no idea how I obtained it.

darkwolves 29-04-2020 10:47

i have suspected the amazing spiderman 2 and the first may use unity i have not tested that thoery yet but i think it may possibly use that


All times are GMT -7. The time now is 18:53.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com