FileForums - Best Compression Methods for 'Specific' Games. Q&A

Pathfinder: Kingmaker Enhancend + Imperial Edition 2.0.8 (35465)

*.bnk, *.wem - srep+razor
*.resS, *.assets - srep+lolz
Bundle folder = xtool(v0.11 zlib+lz4)+srep+lolz

45 gb > 15.1 gb

Compression time - 11 hours
Decompression time - 35 mins

Has anyone been able to get the lz4 xtool plugin working with this title? No matter the version or combination of methods I always seem to get decompression errors when packing with lz4. Tried a handful of other libraries but doesnt seem to help.

KaktoR

20-09-2021 00:55

@L33THAK0R
I don't get it to work either, but for me the end is right in the begining of compression no matter what library I use :D
"ERROR: general (de)compression error in xprecomp"

Edit: Now I used xtool 0.3.16H + unity plugin + lz4 v180 library and it worked fine. Will test some more.

I don't know if the unity plugin is compatible with latest xtool version?

Edit 2: Ok unity plugin works with latest xtool version 0.3.21

Code:

newdunswardoutdoor_light_night_bundle



Compressed 1 file, 124,807,756 => 219,835,242 bytes. Ratio 176.14%

Compression time: cpu 0.11 sec/real 3.73 sec = 3%. Speed 33.42 mB/s

All OK



Extracted 1 file, 219,835,242 => 124,807,756 bytes. Ratio 176.14%

Extraction time: cpu 0.03 sec/real 3.16 sec = 1%. Speed 39.54 mB/s

All OK

L33THAK0R

20-09-2021 01:32

Quote:

Originally Posted by KaktoR (Post 494061)

Code:

newdunswardoutdoor_light_night_bundle



Compressed 1 file, 124,807,756 => 219,835,242 bytes. Ratio 176.14%

Compression time: cpu 0.11 sec/real 3.73 sec = 3%. Speed 33.42 mB/s

All OK



Extracted 1 file, 219,835,242 => 124,807,756 bytes. Ratio 176.14%

Extraction time: cpu 0.03 sec/real 3.16 sec = 1%. Speed 39.54 mB/s

All OK

This is quite the relief that I wasn't the only one having issues! Would it be possible for you to pretty please upload the v180 lz4 library? I can't seem to find it online (my collection of lz4 libraries is only the different ones included in Razor's xtool builds) unfortunately.

Masquerade

20-09-2021 01:54

Quote:

Originally Posted by L33THAK0R (Post 494062)

Would it be possible for you to pretty please upload the v180 lz4 library?

https://github.com/lz4/lz4/releases/tag/v1.8.0

KaktoR

20-09-2021 01:54

1 Attachment(s)

https://github.com/lz4/lz4/releases/tag/v1.8.0

L33THAK0R

20-09-2021 02:45

I had no idea this repository existed! Thank you both for providing links, this should be quite useful in future testing!

Carldric Clement

23-09-2021 03:32

Slime Rancher v1.4.3

Original Size: 1.13 GB
Compressed Size: 467 MB

Options:

Code:

Rest of files: srep:m3f:a2+exe2+lolz



Lolz Setting: -d128m -mt4 -mtt1 -lmdf1

Masquerade

23-09-2021 04:37

Carldric Clement
It's just a unity game, xtool isn't needed. Nothing for it to precompress.

L33THAK0R

24-09-2021 21:10

Does anyone know what "Oodle" algorithims "Planet Zoo" uses? I'm fairly sure it uses "Kraken" (output size was inflated a considerable amount when testing with the "Kraken" xtool plugin) but I'm unsure if additional algorithims are used. The scanner tool thingamajig outputs a "0" for all files (unknown/invalid).

mr_gnar

29-09-2021 18:19

NEO: The World Ends With You
srep+lzma (Without staff roll videos)

Output size: 11.1 GB

8+ GB worth of USMs PLUS encrypted files...not worth messing with this unless you plan to re-encode them, and I don't even know if this one of those games where it has issues with touched USMs.

HunterEmrah

01-10-2021 09:41

can someone please share actual method for Total War Warhammer II with rebuilding pack files?

dixen

01-10-2021 10:06

Quote:

Originally Posted by HunterEmrah (Post 494193)

can someone please share actual method for Total War Warhammer II with rebuilding pack files?

No actual. with rebiulded pack new version TTWH2 - don't work

KaktoR

02-10-2021 05:08

Total War: Warhammer II
v1.12.0 incl all DLC

Code:

 10:57:36 - Selected ARC/DS method for Data1a-01.bin was: xzlib+srep+LZMA-MT

----------------------------------------------------------------------------------------------------------------------------------------

 13:52:22 - Overall input size: 64.01 GB

 13:52:22 - Overall output size: 38.75 GB (Ratio 60.54%)

 13:52:22 - Data files creation time: 02:30:41

xtool ratio is about 106%. Not much, but that's in general in TW games.

movies contain vp8 files, sound wem.

panker1992

02-10-2021 11:13

Quote:

Originally Posted by KaktoR (Post 494204)

Total War: Warhammer II
v1.12.0 incl all DLC

Code:

 10:57:36 - Selected ARC/DS method for Data1a-01.bin was: xzlib+srep+LZMA-MT

----------------------------------------------------------------------------------------------------------------------------------------

 13:52:22 - Overall input size: 64.01 GB

 13:52:22 - Overall output size: 38.75 GB (Ratio 60.54%)

 13:52:22 - Data files creation time: 02:30:41

xtool ratio is about 106%. Not much, but that's in general in TW games.

movies contain vp8 files, sound wem.

Your method is incomplete, this game uses LZMA compression by their devs
you can REMOVE it with RPFM.

KaktoR

02-10-2021 11:55

Quote:

Originally Posted by panker1992 (Post 494220)

Your method is incomplete, this game uses LZMA compression by their devs
you can REMOVE it with RPFM.

Yes, but it would lead to bad crc and I don't know if the game works fine with not matching original crc.

dixen

02-10-2021 23:53

Quote:

Originally Posted by panker1992 (Post 494220)

Your method is incomplete, this game uses LZMA compression by their devs
you can REMOVE it with RPFM.

On rebuilded terrain.pack - crashes Battle mode.

macut18

03-10-2021 04:39

Them and Us
uelr+srep+4x4 = 25gb >> 7gb
- bpk on bik videos

@Thank KaktoR

KaktoR

03-10-2021 05:14

Use uelr on *.tfc|*.u|*.udk|*.upk files

You can use bpk on bik videos.

KaktoR

03-10-2021 05:28

Anno 1800
v12.1, all DLC, nothing ripped

Code:

 17:00:46 - Selected ARC/DS method for Data1a-01.bin was: srep_new+razor_mtx

 17:00:46 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000

--------------------------------------------------------------------------------------

 21:19:20 - Overall input size: 68.95 GB

 21:19:20 - Overall output size: 28.63 GB (Ratio 41.52%)

 21:19:20 - Data files creation time: 03:59:40

hydefromt70s

04-10-2021 14:22

Quote:

Originally Posted by Masquerade (Post 489501)

uhhhh ok
2. Why are you using the cyberpunk 2077 precompressor on not cyberpunk 2077?????? :confused::confused::confused:

3. xzlib - we're getting somewhere, but this game is made in unreal engine 3 and is compressed with lzo and not zlib (zlib is ue4) - which is why you need to use UELR [Unreal Engine LZO Recompressor]

I know you write this post long time ago but just to clarify something. Cyberpunk 2077 wasn't made in UE3 as you said, it was made in CDProject's REDengine 4.

L33THAK0R

04-10-2021 17:17

Quote:

Originally Posted by hydefromt70s (Post 494257)

I know you write this post long time ago but just to clarify something. Cyberpunk 2077 wasn't made in UE3 as you said, it was made in CDProject's REDengine 4.

The title in question was "Rocket League" though, which was made in "Unreal Engine 3" and not "CDProjekts REDengine 4".

KaktoR

06-10-2021 01:03

Do not use bpk on Alan Wake Remastered. Pretty much all videos give bad crc on extracting. I don't know if bpk has problems with 4K resolution or codec they use.

neko	06-10-2021 03:26

Quote:

Originally Posted by KaktoR (Post 494283)

Do not use bpk on Alan Wake Remastered. Pretty much all videos give bad crc on extracting. I don't know if bpk has problems with 4K resolution or codec they use.

Thanks for clarification. I have been confused and trying to figure what went wrong for the whole day :(

kj911

08-10-2021 07:58

Noob questions: This ECM prefilter tested from very old/older game files (like: textures, audio, any formats) compressions? Use before any precompressors or before srep+lzma/lolz compressions? Yes, the filter optimized this CD-image files compression.

https://encode.su/threads/2829-RAZOR...ll=1#post54307
https://encode.su/threads/2829-RAZOR...ll=1#post54336

st0rm

08-10-2021 09:13

Alpha Prime [2007]

Code:

Game Version : 1.3



Ripped - Re-encoded Files : Nothing



Original Size : 978 MB



Compressed Size : 748 MB



Install Time (Depends your System): 2-3 Minutes



Method : pzlib3+srep:m3f+lzma



Note(s) : Used Pzlib because game is older and low specs friendly. :cool:

FitGirl

09-10-2021 09:36

Quote:

Originally Posted by kj911 (Post 494314)

ECM is a special filter, which removes/restores special ECC/CRC codes from CD ISOS. It's useless on anything else but ISOS with ECC codes.

:( Sad8669

12-10-2021 10:27

Farcry 6 - Precompression

Library used : LZ4 v1.8.2 (Thanks to KaKtor)
Similar results can be obtained from LZ4 that comes with XTool v0.3.9.
XTool Version : 0.3.9
Database : Created using Razor's db (fc6.xtl)

Code:

[External compressor:xtool]

header    = 0

packcmd   = xtool.exe precomp -mlz4+fc6 -c32mb -t100p --dbase - - <stdin> <stdout>

unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

Code:

Compressed 1 file, 683,722,320 => 1,286,884,486 bytes. Ratio 188.22%

Compression time: cpu 1.02 sec/real 48.29 sec = 2%. Speed 14.16 mB/s

All OK

lomesmo

16-10-2021 11:51

A Hat In Time - Ultimate Edition

All DLC

Code:

Input size: 8.37 GB

Output size: 3.88 GB

Method: xLZO+srep+exe+7z+lzma2

lomesmo

16-10-2021 11:53

AKIBA'S TRIP - Hellbound & Debriefed

Code:

Input size: 2.47 GB

Output size: 1.33 GB

Method: xZLib+srep+7z+lzma2

lomesmo

16-10-2021 13:48

Bullet Witch

All DLC

Code:

Input size: 4.57 GB

Output size: 3.09 GB

Method: xZLib+srep+lzma2

L33THAK0R

16-10-2021 18:28

Hi all,

Has anyone given "Call of Duty: Black Ops III (3)" a shot? It piqued my interest when I saw the QuickBMS script for "xpack" archives [1] mentioned that the "LZO", "LZ4" & "Oodle" compression algorithms were used within these archives. I'm sure testing which libraries for "LZ4" give the best output won't be too tricky, but in regards to the "Oodle" codec(s) used, I'm a bit stumped. As far as I know a "oo2core_#_win64.dll" is required, with either "xtool" or "Oo2Rec#", to pre-compress a given file, however such a file isn't present within the root and sub-directories of the title in question. Does anyone have any idea how to go about pre-compressing the "Oodle" streams for this title? Apologies if I buggered the terminology.

KaktoR

17-10-2021 03:27

Well you can check with hex edit if the game has oodle at all (I don't know), if yes then just try all different libraries.

kj911

17-10-2021 14:32

Quote:

Originally Posted by kj911 (Post 491663)

WAVE compression again: Retested few switches, use TinyGamePack tools by Panker1992 and from searching best results.

In last time good results: 635 350kB
....

Newest results, from previous months testing:

MSC+SREP: 634 861kB (Use TAK 2.3.0)
MSC+SREP: 634 836kB (Use TAK 2.3.1)

Retested and again use MSC with TAK 2.3.1 and use with any extra switches and goal!!

New Results: 630 595kB!! Nice, few MB's gain from previous versions. Overmaxed WAV compreesion. :)

Tiny SFX compressor commands:

Code:

arc a -lc1024 -ld1024 -ep1 -ed -r -w.\ data.bin -mmsc+rep:256mb+4x4 "Data\*"

arc.ini:

Code:

[External compressor:msc]

header = 0

packcmd = MSC c -v -f -wav=1 -raw=2 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=2 -bmf=9s -tak=9 -dxt=2  $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

CRCs are OK! My preference, smallest memory eating, (lower than 768MB/1GB) from during decompress, heavy compressed, repacked games to use old PC's. (Win XP with min. 1-1.25GB RAM or Win7 with 2GB RAM memory.)

More screens:

https://i.kek.sh/MaGjls4ga1X.jpg
https://i.kek.sh/468nxWxsqpv.jpg
https://i.kek.sh/JK1wgOZUDQF.jpg
https://i.kek.sh/9kw96quv88w.jpg

KaktoR

21-10-2021 02:41

NieR Replicant ver 1 22474487139

Anyone knows what zstd lib to use? It's slow af

Masquerade

21-10-2021 06:26

Quote:

Originally Posted by KaktoR (Post 494554)

NieR Replicant ver 1 22474487139

Anyone knows what zstd lib to use? It's slow af

Unsure, I have checked with latest ZSTD to see if it can list info about the archives, yet it can only seem to load info for few archives. Yet, all files are definitely compressed with zstd.

Code:

for %f in (*.arc) do zstd -l "%f"

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

     1      0     9.25 MB      30.19 MB  3.263   None  common.arc

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

     1      0   690.86 KB    1748.36 KB  2.531   None  info.arc

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

File "init.arc" not compressed by zstd

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

File "lang1.arc" not compressed by zstd

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

File "lang2.arc" not compressed by zstd

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

File "param.arc" not compressed by zstd

Frames  Skips  Compressed  Uncompressed  Ratio  Check  Filename

File "stream.arc" not compressed by zstd

I even tried compressing a demo file with zstd and testing the latest libzstd to see if the precompressor worked, neither zstdrec or xtool could precompress a random file compressed with latest zstd.

KaktoR

21-10-2021 06:46

Yes, the first file contains just 1 block with zstd (28B52FFD).

However, ratio with lzma2 for init.arc is about 70%.

I will investigate it further :D

Masquerade

23-10-2021 02:53

KaktoR
You can use the following tools to extract all of the compressed ZSTD streams from NieR replicant. It will allow for decompression of 99% of streams.

1. VGM Toolbox and this script. Download latest zstd.exe.

2. Place the XML file in plugins\AdvancedCutter folder of VGMToolbox.

3. Open VGMToolbox, go to Misc. Tools > Extraction Tools > Generic > Advanced Cutter

4. From the drop down menu in the top right, select NieRReplicant and click LOAD.

5. Next, you can drop each of the arc files into the top left box and it will extract the streams. You only really need to do this for the large arc files. No need on the little ones.

6. After extracting the streams, run this python script for each of the files. This will remove the extra padding bytes (@) that the devs added so that each stream begins at an offset of divisible by 16. ZSTD will not decompress files that contain data after the end of the stream.

Code:

for %f in (*.MSQ) do trimmer.py "%f"

Ignore that it involves .MSQ extension. You can switch it to .TMP or whatever.

7.

Code:

for /r %f in (*.MSQ) do zstd -d "%f" -o "%~df%~pf%~nf.DEC"

This will use zstd to decompress the streams. Afair it fails on about 16 streams out of ~7800, but you can decompress most of the game this way.

KaktoR

23-10-2021 06:10

Question is if it's profitable because with plain srep+lzma i get 13.5GB

All times are GMT -7. The time now is 11:02.

Page 65 of 119

« First

115

Last »

Show 40 post(s) from this thread on one page