PDA

View Full Version : Best Compression Methods for 'Specific' Games. Q&A


Pages : 1 2 3 4 5 6 7 8 9 10 [11] 12 13 14 15 16 17 18 19

panker1992
16-07-2021, 09:52
what you saw is high quality 400kbps all the way to low quality 128kbps

i save many GB by doing this :P

L33THAK0R
16-07-2021, 10:15
what you saw is high quality 400kbps all the way to low quality 128kbps

i save many GB by doing this :P

I look forward to being able to share some similarly impressive results in the near future!

Masquerade
17-07-2021, 10:34
Start Size: 60.4GB

xreflate+srep+lolz

For bonus content:

precomp+srep+lolz

Final Size: 29.2GB (29.4GB with Bonus Content)

This took 18-19 hours for me in total.

L33THAK0R
17-07-2021, 21:13
UPDATE: It seems it was an error on my end after-all.

Has anyone else had issues with reflate (https://imgur.com/a/WzQP6iO) & preflate (https://imgur.com/a/zdbeMvE) regarding "The Saboteur"? I ran reflate for a good 14hrs prior, as a first test, but alas it was still stuck at the same spot. Just want to know if it's something on my end or just the title itself, as the "Odin Engine" is a custom game-engine only used for this title (I don't know if its a fork of something like UE3 or Source, I couldn't find any documentation on it, after a brief searching period), and I'm not sure how its game-data archives function.

Masquerade
18-07-2021, 02:12
UPDATE: It seems it was an error on my end after-all.

Has anyone else had issues with reflate (https://imgur.com/a/WzQP6iO) & preflate (https://imgur.com/a/zdbeMvE) regarding "The Saboteur"? I ran reflate for a good 14hrs prior, as a first test, but alas it was still stuck at the same spot. Just want to know if it's something on my end or just the title itself, as the "Odin Engine" is a custom game-engine only used for this title (I don't know if its a fork of something like UE3 or Source, I couldn't find any documentation on it, after a brief searching period), and I'm not sure how its game-data archives function.

Try using old xtool. Also use stdio instead of using $$arc$$s, it's faster. Also, srep+lolz+bpk makes bpk useless since any bink data will be put through srep and lolz.

KaktoR
18-07-2021, 03:22
Someone knows a good compressor for gif images? If possible.

L33THAK0R
18-07-2021, 04:34
Try using old xtool

Yeah ended up going with xtool 2019.

Also use stdio instead of using $$arc$$s, it's faster

I'm still a massive newbie so got no clue what stdio is, I'll have to research this

Also, srep+lolz+bpk makes bpk useless since any bink data will be put through srep and lolz.

I'm fully aware, the actual order of compressors for that preset is bpk + srep + lolz, but as its my own script I haven't seen the need to change that (I have ~40 copies of it on my system, spread across multiple drives to limit the impact of storage I/O limitations, making it a pain to update all of em since at least 7 instances are running at any given time). Now that I think about it I used to use a mask for the bpk compressor (srep+lolz/$BINK=bpk) until a few months back when I encountered bink video files under different extension names & within archives, and stopped using it as a mask (couldn't be bothered constantly updating the mask), which probably explains the label of the preset.

BKR-TN
18-07-2021, 04:56
Someone knows a good compressor for gif images? If possible.
You maybe want to take a look at this:
flexiGIF - Lossless GIF/LZW optimization (https://create.stephan-brumme.com/flexigif-lossless-gif-lzw-optimization/)
Keep in mind that flexiGIF's compression is very (!) slow, magnitudes slower than a standard GIF encoder.

KaktoR
18-07-2021, 05:54
You maybe want to take a look at this:
flexiGIF - Lossless GIF/LZW optimization (https://create.stephan-brumme.com/flexigif-lossless-gif-lzw-optimization/)

Very slow won't describe on how slow it actually is, not at all. Holy sh...

Masquerade
18-07-2021, 06:31
Very slow won't describe on how slow it actually is, not at all. Holy sh...

Precomp supports gifs, run precomp+srep+your choiceof compress

Masquerade
18-07-2021, 06:34
I'm still a massive newbie so got no clue what stdio is, I'll have to research this

Instead of $$arcdatafile$$.tmp $$arcpackedfile$$.tmp,you use - - <stdin> <stdout>

Example for old xtool:

packcmd = xtool e:precomp:t100p,c128m:zlib - - <stdin> <stdout>

L33THAK0R
18-07-2021, 08:35
Instead of $$arcdatafile$$.tmp $$arcpackedfile$$.tmp,you use - - <stdin> <stdout>

Example for old xtool:

packcmd = xtool e:precomp:t100p,c128m:zlib - - <stdin> <stdout>

Ah alright, I didn't think there was an actual performance difference between the two, I'll run some tests to see what the difference is!

panker1992
19-07-2021, 04:28
packcmd = xtool e:precomp:t100p,c128m:zlib - <stdin> $$arcpackedfile$$.tmp

according to my endless debug with Zee this one is the best way to avoid some problems

the file will have to be copied only once before srep actually takes it in. but its best in terms of debuggery

:( Sad8669
20-07-2021, 11:07
Method : srep:m3f:l512+lolz:dtb1:d128:mtt0:mt4:mc1023
Compression Time : 11 Minutes 4 Seconds.
Decompression Time : 6 Seconds.
Initial Size : 716.68 MB
Final Size : 109.81 MB

Masquerade
23-07-2021, 03:38
Start size: 47.7GB

xzlib+srep+lolz

Soundtrack: mpz

Compressed size: 29.2GB

Takes around ~15 hours.

x86 ripped.

KaktoR
23-07-2021, 03:40
Start size: 47.7GB

xzlib+srep+lolz

Soundtrack: mpz

Compressed size: 29.2GB

Takes around ~15 hours.

xzlib+srep+4x4:lzma
30.4 GB

Takes around ~2 hours

PS: Ripped x86 stuff (Data_LE, Game_LE)

Masquerade
23-07-2021, 03:45
Takes around ~2 hours
https://i.ibb.co/Yk9gM3s/I8Iay5H.jpg
+13 hours for -1.2GB :p

:( Sad8669
23-07-2021, 04:01
The pain.....

Although, two methods are appreciated in the INDEX :P.

KaktoR
23-07-2021, 04:10
https://i.ibb.co/Yk9gM3s/I8Iay5H.jpg
+13 hours for -1.2GB :p

Not a good time/output ratio if you ask me :D But I'm not a repacker who share his stuff :p

Therefore, for me 4x4:lzma is the best in time/output ratio in most cases. I just use lolz for testing stuff anyway.

:( Sad8669
24-07-2021, 11:26
Is there a way to compress .awb audio files?

1:1 CRC Perfect.

Masquerade
25-07-2021, 02:05
Is there a way to compress .awb audio files?

1:1 CRC Perfect.

No, AWB is a lossy format so even when you extract the wav audio, there's no way you can patch it back.

:( Sad8669
26-07-2021, 06:15
Edit : Nothing.

Masquerade
27-07-2021, 06:12
xzlib+srep+lolz

27.2GB -----> 13.2GB

Masquerade
27-07-2021, 09:38
xzlib+srep+lolz
for movie folder, use
srep+lzma
15.1GB ---> 9.6GB

MV059.bin and MV060.bin are credits videos, so you can rip them to give you a final of 9.1GB.

KaktoR
28-07-2021, 11:36
The Forgotten City

19:37:28 - Selected ARC/DS method for Data1a-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
20:35:24 - Overall input size: 25.84 GB
20:35:24 - Overall output size: 9.43 GB (Ratio 36.50%)
20:35:24 - Overall conversion time: 00:57:55


19:28:10 - Selected ARC/DS method for Data1a-01.bin was: m1
----------------------------------------------------------------------------------------------------------------------------------------
19:36:38 - Overall input size: 25.84 GB
19:36:38 - Overall output size: 12.46 GB (Ratio 48.20%)
19:36:38 - Overall conversion time: 00:08:25

dixen
28-07-2021, 12:05
Outlast 2 (GOG) (English voice only)

*.TFC, *.UPK - uelr+srep+lolz
23,2 gb > 36 gb > 24 gb > 11,6 gb

*.bnk, *.ol2 - srep+lzma2
1,7 gb > 1,3 gb

Rest files - srep+lolz
115 mb > 55 mb

Repack size - 13 gb
Install size - 25 gb
Install time - 4 mins. (r5 3600, HDD)

dixen
29-07-2021, 06:18
Crysis 3 (DDE) (RUS+ENG)

*.pak - XTool (with CRYEngine plugin)+SREP+LOLZ
10.8 gb > 16.5 gb > 11 gb > 6.9 gb

Rest files - SREP+LOLZ
43 mb > 13 mb

packcmd = xt21\xt21 precomp -mcrysis3+zlib+preflate -d1 -c256mb -t100p - - <stdin> <stdout>

:( Sad8669
30-07-2021, 06:09
Naruto - Ultimate Ninja Storm - Precompression


dataRegion.cpk
Compressed 1 file, 436,892,316 => 1,358,762,846 bytes. Ratio 311.01%
Compression time: cpu 1.22 sec/real 26.37 sec = 5%. Speed 16.57 mB/s
All OK

Codec : Crilayla

Some .cpk like movie and sound contains no streams, so you can save time by separating them.

:( Sad8669
01-08-2021, 08:06
Method : xtool_crilayla+srep:m3f:l512+lolz:dtb1:d128:mtt0:m t4:mc1023
Compression Time : 2 Hours 21 Minutes 28 Seconds
Decompression Time : 9 Minutes 10 Seconds
Initial Size : 7.53 GB
Final Size : 2.43 GB
Ripped Files : Europe - Japanese Movies + JP Sound Files + Credit Video
Re-encoded : All Main Movies.

-----------------------
Movie File | Size : 2.02 GB | Blocks/Offsets
-----------------------
opening_EU.usm ---> 3586646016 - 3776301631
opening_JP.usm ---> 3776303104 - 3965540352
da1050_JP.usm ---> 2699368448 - 3310075263
da1049_JP.usm ---> 1485172736 - 2097358367
da1001_JP.usm ---> 596582400 - 872960095
da1001_EU.usm ---> 298297344 - 596580511
da1001.usm ---> 4096 - 298295615
------------------------

Note : Game is tested and is playable, but before you start the match you must select the sound which you didn't rip out otherwise it will crash the game.


Tools Used + Thanks to : - QuickBms - HxD Hex Editor - Scaleform Encoder - VGMTool Box - Masquerade - panker1992 - Everyone

panker1992
01-08-2021, 13:05
Method : xtool_crilayla+srep:m3f:l512+lolz:dtb1:d128:mtt0:m t4:mc1023
Compression Time : 2 Hours 21 Minutes 28 Seconds
Decompression Time : 9 Minutes 10 Seconds
Initial Size : 7.53 GB
Final Size : 2.43 GB
Ripped Files : Europe - Japanese Movies + JP Sound Files + Credit Video
Re-encoded : All Main Movies.

-----------------------
Movie File | Size : 2.02 GB | Blocks/Offsets
-----------------------
opening_EU.usm ---> 3586646016 - 3776301631
opening_JP.usm ---> 3776303104 - 3965540352
da1050_JP.usm ---> 2699368448 - 3310075263
da1049_JP.usm ---> 1485172736 - 2097358367
da1001_JP.usm ---> 596582400 - 872960095
da1001_EU.usm ---> 298297344 - 596580511
da1001.usm ---> 4096 - 298295615
------------------------

Note : Game is tested and is playable, but before you start the match you must select the sound which you didn't rip out otherwise it will crash the game.


Tools Used + Thanks to : - QuickBms - HxD Hex Editor - Scaleform Encoder - VGMTool Box - Masquerade - panker1992 - Everyone



It seems that you have gotten the idea !!! interesting

i would like to see future repacks made this way, as i find it very resource wasting to have videos 30mbps and audio of the highest quality with no real reason

KaktoR
13-08-2021, 01:39
Assassin's Creed Valhalla
v1.3.0, 1 language, all DLCs

23:23:14 - Selected ARC/DS method for Data1a-01.bin was: oo2recm7+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb2 73:lc8:mc1000000
23:23:14 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
--------------------------------------------------------------------------------------
23:23:18 - Starting Data1a-01.bin archive creation.
09:20:36 - Moving Data1a-01.bin file to media folder.
09:20:40 - Moving Data1a-02.bin file to media folder.
09:20:40 - Starting Data1b-01.bin archive creation.
09:32:41 - Moving Data1b-01.bin file to media folder.
--------------------------------------------------------------------------------------
10:04:24 - Overall input size: 104,93 GB
10:04:24 - Overall output size: 38,42 GB (Ratio 36.62%)
10:04:24 - Data files creation time: 10:09:27

Masquerade
15-08-2021, 02:01
Start size: 1.7GB

precomp:cn:brute:d1+srep+lolz

Final Size: 1GB

KaktoR
18-08-2021, 00:25
Grand Theft Auto V
v1.0.231.0 / 1.54

01:08:30 - Selected ARC/DS method for Data1a-01.bin was: bpk+xtool:mreflate+srep_new+4x4:b64mb:lzma:ultra:6 4m:bt4:fb273:lc8:mc1000000
01:08:30 - Selected ARC/DS method for Data1b-01.bin was: xtool:mreflate+srep_new+4x4:b64mb:lzma:ultra:64m:b t4:fb273:lc8:mc1000000
01:08:30 - Selected ARC/DS method for Data1c-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
05:39:13 - Overall input size: 102.55 GB
05:39:13 - Overall output size: 62.48 GB (Ratio 60.93%)
05:39:13 - Data files creation time: 04:30:42

Data1a contents:
update\update.rpf
update\x64\dlcpacks\patchday3ng\dlc.rpf
update\x64\dlcpacks\patchday17ng\dlc.rpf
update\x64\dlcpacks\patchday19ng\dlc.rpf
update\x64\dlcpacks\patchday20ng\dlc.rpf
update\x64\dlcpacks\patchday21ng\dlc.rpf
update\x64\dlcpacks\patchday22ng\dlc.rpf
update\x64\dlcpacks\patchday24ng\dlc.rpf
update\x64\dlcpacks\patchday25ng\dlc.rpf
x64a.rpf
x64d.rpf

Data1b contents:
*.rpf
exclude Data1a contents and x64\*

Data1c contents:
rest of the files

KaktoR
18-08-2021, 02:16
HUMANKIND

10:24:57 - Selected ARC/DS method for Data1a-01.bin was: srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
11:15:53 - Overall input size: 27.22 GB
11:15:53 - Overall output size: 10.20 GB (Ratio 37.48%)
11:15:53 - Overall conversion time: 00:50:53

KaktoR
20-08-2021, 00:31
Cyberpunk 2077
v1.30

22:43:05 - Selected ARC/DS method for Data1a-01.bin was: xtool:c32mb:mcp2077+srep_new+4x4:b64mb:lzma:ultra: 64m:bt4:fb273:lc8:mc1000000
22:43:05 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
--------------------------------------------------------------------------------------
05:54:53 - Overall input size: 66.50 GB
05:54:53 - Overall output size: 43.73 GB (Ratio 65.76%)
05:54:53 - Data files creation time: 06:44:23

Data1a contents:
*.archive

Data1b contents:
rest of the files

Masquerade
20-08-2021, 13:32
precomp+srep+lolz/$ogg=oggre

1.4GB ---> 370MB

Masquerade
20-08-2021, 13:33
XZLIB+SREP+LOLZ

46.6GB ---> 22.3GB

I used -d3 param with zlib+reflate.

Masquerade
20-08-2021, 13:33
xzlib+srep+lolz

14.9GB ---> 5.5GB

Masquerade
20-08-2021, 13:34
srep+lolz

3.5GB ---> 1.1GB

Masquerade
20-08-2021, 13:35
xzlib+srep+lolz

4.9GB ---> 3GB

Masquerade
21-08-2021, 09:21
xzlib+srep+lolz

Now, this is a unity game and you only need srep and lolz, however SREP would crash when trying to run on this game files, so I put xtool in front and it worked (failure on both srep 3.92b and srep 3.2). XTool does nothing to actually improve compression here.

18.2GB ---> 3.8GB

L0v3craft
22-08-2021, 00:48
Mortal Shell [Build 7180006]

xzlib+srep+lolz

14.9GB ---> 5.5GB

Please write the full parameters that you used for lolz and which version of xtool have you used.

KaktoR
22-08-2021, 10:30
Dying Light Platinum Edition

16:27:24 - Selected ARC/DS method for Data1a-01.bin was: bpk+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc 8:mc1000000
16:27:24 - Selected ARC/DS method for Data1b-01.bin was: xtool:mreflate+srep_new+lzma:a1:mfbt4:256m:fb273:m c1000000000:lc8
----------------------------------------------------------------------------------------------------------------------------------------
19:00:21 - Overall input size: 35.08 GB
19:00:21 - Overall output size: 17.70 GB (Ratio 50.44%)
19:00:21 - Overall conversion time: 02:32:55


Data1b contents:
*.pak, *.rpack

Data1a contents:
rest of the files

mr_gnar
24-08-2021, 10:14
Has anyone tried Serious Sam 3 (BFE)? I have tried the method on here (by dixen), but for some reason, it doesn't work as well.

Xtool (zLib codec)+msc+srep+lolz - 6 gb > 3.3 gb

6 GB > 5.55GB for me. I tried changing out my version of XT and srep/lolz as well but it gets almost the same result. :confused:

Masquerade
24-08-2021, 10:33
Has anyone tried Serious Sam 3 (BFE)? I have tried the method on here (by dixen), but for some reason, it doesn't work as well.



6 GB > 5.55GB for me. I tried changing out my version of XT and srep/lolz as well but it gets almost the same result. :confused:

Make sure you don't have LZMA enabled in your MSC configs, since it will compress everything input with LZMA after any media streams.

KaktoR
26-08-2021, 08:32
Small language update for Spyro Reignited Trilogy (if anyone cares at all :D).

Was able to figure out all languages by Audio Track ID

ID Language
1 English
2 Arabic
3 Danish
4 German
5 Spanish Mexico *
6 Spanish Spain *
7 Finnish
8 French
9 Italian
10 Dutch
11 Norwegian
12 Polish
13 Brazil **
14 Swedish

*Example: Spoken "[...]es una criatura simplona."
But I'm not sure what is Spain and Mexico. We need some guidance from Spain/Mexican people here :D

**In Portuguese it is "[...]é uma criatura simples."

I will do this same shit for Crash Bandicoot.

KaktoR
26-08-2021, 14:30
Language update for Crash Bandicoot 4 It's about Time

ID Language
1 English
2 Arabic
3 German
4 Spanish Mexico ???
5 Japanese
6 Spanish
7 French
8 Italian
9 Portuguese Brazil

I couldn't figure out what language ID 4 is yet.

PS: ID 4 is somewhat spanish mexico or whatever. They say "si" for "yes". But other keywords don't match with spanish (spain) or italian.

lomesmo
26-08-2021, 18:04
es una criatura simplona."
But I'm not sure what is Spain and Mexico. We need some guidance from Spain/Mexican people here :D

**In Portuguese it is "[...]é uma criatura simples."

I will do this same shit for Crash Bandicoot.

Spanish (Mexico) is Spanish for Latin America. Spanish (Spain) is European, the difference lies in words and adjectives.

For example, mobile phone (Spain) and cell phone (Latin America), things like that

Telefono movil (España) - Celular (Latinoamerica)(Mexico)

lomesmo
26-08-2021, 18:08
It is essentially the same only that in Latin America and in Spain different words are used to express the same idea

KaktoR
27-08-2021, 09:23
Makes sense, but because I don't speak spanish I can't tell you any deeper differences between Spanish and Latam ;)

KaktoR
30-08-2021, 11:39
xtool+srep+lolz
On *.webm and *.feral
srep+lzma2
*feral is ogg audio archives, so you could use oggre but no idea of the benefit (or any issue in decompression)

50.1GB ----> 17.3GB :D

You can rip languages from webm videos with ffmpeg (thanks to devs streams are labeled).

Only 1 language: 1.78 GB -> 1.68 GB

You can use MSC on the following files
Contents\Resources\Data\data\sounds\sfx.dat.feral

Contents\Resources\Data\data\sounds\localised\de\v oice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\de\v oice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\en\v oice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\en\v oice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\it\v oice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\it\v oice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\es\v oice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\es\v oice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\ru\v oice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\ru\v oice_strat.dat.feral

Contents\Resources\Data\data\sounds\localised\fr\v oice_battle.dat.feral
Contents\Resources\Data\data\sounds\localised\fr\v oice_strat.dat.feral

KaktoR
31-08-2021, 05:59
Total War Rome Remastered
v2.0.2, 1 language, other languages ripped, no hd pack

10:53:41 - Selected ARC/SPLIT method for Data1a-01.bin was:
srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
$deflatepack=xtool:mreflate+srep_new+4x4:b64mb:lzm a:ultra:64m:bt4:fb273:lc8:mc1000000
$mscromeremastered=msc_tak:mp3=1:wav=1:raw=1:ddsra w=1:bmf=9:TAK=9:bmp=1:lzma,bt4,fb128,lc8
$textpack=srep_new+lzma:a1:mfbt4:256m:fb273:mc1000 000000:lc8
$precomppack=precomp_mtx+srep_new+lolz:mtt1:mt<thread>:tt1:mc2
$libpack=srep_new+razor_mtx
----------------------------------------------------------------------------------------------------------------------------------------
13:29:13 - Overall input size: 43.22 GB
13:29:13 - Overall output size: 19.46 GB (Ratio 45.02%)
13:29:13 - Overall conversion time: 02:27:30

$deflatepack: *.png
$msc: see post above
$textpack: *.txt, *.xml, *.json, *.strings
$precomppack: *.gif
$libpack: *.dll, *.exe

video files for ffmpeg: Contents\Resources\Data\data\fmv\intros (1 language: 1.78 GB -> 1.68 GB)

KaktoR
03-09-2021, 01:10
Far Cry 5
v1.011, 2 languages, incl HD pack

21:41:34 - Selected ARC/DS method for Data1a-01.bin was: xtool:mreflate+srep_new+4x4:b64mb:lzma:ultra:64m:b t4:fb273:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
02:20:26 - Overall input size: 91.60 GB
02:20:26 - Overall output size: 39.95 GB (Ratio 43.61%)
02:20:26 - Overall conversion time: 04:38:50

Used DELZORec_x64 (https://fileforums.com/showpost.php?p=493768&postcount=4) because xtool with fc5 plugin does not work properly atm.

DELZORec times:
8min ssd
25min hdd

PS: DELZORec does not work on any hd game files (*_hd.*)

DELZORec usage from batch (common.dat example):
DELZORec_x64.exe u "data_final\pc\common.fat" "data_final\pc\common.dat" "data_final\pc\common.unp"
DELZORec_x64.exe r "data_final\pc\common.unp" "data_final\pc\common.dat" "data_final\pc\common.fat"

hwang4
06-09-2021, 12:07
What's Best Method for compression?

Masquerade
06-09-2021, 13:41
What's Best Method for compression?

This game's PKG archives are compressed using LZ4, so first you need to find a way to decompress files inside the archives. Since the header of the files appear to be encrypted, unlike previous games, there's no BMS script to unpack the PKG.

hwang4
07-09-2021, 06:13
Masquerade , can you show me your method plz !:)

I tried (xtool+srep+lolz), i got 25,5gb (english+french)!
Your method is better (21gb)
thx!

Masquerade
07-09-2021, 13:29
Masquerade , can you show me your method plz !:)

I tried (xtool+srep+lolz), i got 25,5gb (english+french)!
Your method is better (21gb)
thx!

I used VGM toolbox to "extract" the PKGs, since every lz4 stream is very easy to detect with the 0x04224D18 header with PKGB before the steam and PKGE after the stream. When each file is extracted, you can use lz4 exe to decompress the extracted lz4 streams, then compress with xtool:zlib+srep+lolz.

Then, to remake the pkgs, you just need to compress each file again individually with the lz4 exe using -9 --no-frame-crc arguments. Then, concatenate the files together (I use bincat for this since it's fast and simple, not sure of native cmd option) and xdelta.

Make your xdeltas before compressing your archive, since you will need to decompress all of the extracted files from the pkgs, compress them, then concatenate them to make a temp file. You can compare the temp file with the original pkg to make a delta.

For the sound files, I used razor.

Hexagon123
09-09-2021, 14:25
I found out that Yoku's Island Express use ZLIB?

dixen
12-09-2021, 14:54
Life Is Strange: Before The Storm (GOG)

*.bytes - xtool (v0.12 + lz4 codec) + srep + lolz
15 gb > 33,6 gb > 15 gb > 4.6 gb

*.usm, *.awb, *.acb - Srep + zstd (cls)
6 gb > 4.6 gb

Rest files - srep + lolz
105 mb > 31 mb

RePack size - 9.3 gb
Install time - 6 min. (R5 3600, 16 gb RAM, HDD>HDD)

NOTE. Why ZSTD? Easy)))

Tested 290 files, 5,035,921,557 => 6,416,827,196 bytes. Ratio 78.48%
Testing time: cpu 17.61 sec/real 17.95 sec = 98%. Speed 357.46 mB/s
All OK

Walker96082
13-09-2021, 12:21
Using FreeArc in Ultra Profile
from 25.7 GB to 7.98 GB:D

Jarmen_Kell
14-09-2021, 00:38
Guys, any idea for "Tales of Arise"?

macut18
14-09-2021, 03:04
NBA 2K22
xtool_zlib+srep+4x4
96.3 GB > 70.2 GB :rolleyes:

L33THAK0R
18-09-2021, 20:00
Pathfinder: Kingmaker Enhancend + Imperial Edition 2.0.8 (35465)

*.bnk, *.wem - srep+razor
*.resS, *.assets - srep+lolz
Bundle folder = xtool(v0.11 zlib+lz4)+srep+lolz

45 gb > 15.1 gb

Compression time - 11 hours
Decompression time - 35 mins

Has anyone been able to get the lz4 xtool plugin working with this title? No matter the version or combination of methods I always seem to get decompression errors when packing with lz4. Tried a handful of other libraries but doesnt seem to help.

KaktoR
20-09-2021, 00:55
@L33THAK0R
I don't get it to work either, but for me the end is right in the begining of compression no matter what library I use :D
"ERROR: general (de)compression error in xprecomp"

Edit: Now I used xtool 0.3.16H + unity plugin + lz4 v180 library and it worked fine. Will test some more.

I don't know if the unity plugin is compatible with latest xtool version?

Edit 2: Ok unity plugin works with latest xtool version 0.3.21

newdunswardoutdoor_light_night_bundle

Compressed 1 file, 124,807,756 => 219,835,242 bytes. Ratio 176.14%
Compression time: cpu 0.11 sec/real 3.73 sec = 3%. Speed 33.42 mB/s
All OK

Extracted 1 file, 219,835,242 => 124,807,756 bytes. Ratio 176.14%
Extraction time: cpu 0.03 sec/real 3.16 sec = 1%. Speed 39.54 mB/s
All OK

L33THAK0R
20-09-2021, 01:32
@L33THAK0R
I don't get it to work either, but for me the end is right in the begining of compression no matter what library I use :D
"ERROR: general (de)compression error in xprecomp"

Edit: Now I used xtool 0.3.16H + unity plugin + lz4 v180 library and it worked fine. Will test some more.

I don't know if the unity plugin is compatible with latest xtool version?

Edit 2: Ok unity plugin works with latest xtool version 0.3.21

newdunswardoutdoor_light_night_bundle

Compressed 1 file, 124,807,756 => 219,835,242 bytes. Ratio 176.14%
Compression time: cpu 0.11 sec/real 3.73 sec = 3%. Speed 33.42 mB/s
All OK

Extracted 1 file, 219,835,242 => 124,807,756 bytes. Ratio 176.14%
Extraction time: cpu 0.03 sec/real 3.16 sec = 1%. Speed 39.54 mB/s
All OK

This is quite the relief that I wasn't the only one having issues! Would it be possible for you to pretty please upload the v180 lz4 library? I can't seem to find it online (my collection of lz4 libraries is only the different ones included in Razor's xtool builds) unfortunately.

Masquerade
20-09-2021, 01:54
Would it be possible for you to pretty please upload the v180 lz4 library?

https://github.com/lz4/lz4/releases/tag/v1.8.0

KaktoR
20-09-2021, 01:54
https://github.com/lz4/lz4/releases/tag/v1.8.0

L33THAK0R
20-09-2021, 02:45
I had no idea this repository existed! Thank you both for providing links, this should be quite useful in future testing!

Carldric Clement
23-09-2021, 03:32
Original Size: 1.13 GB
Compressed Size: 467 MB

Options:Rest of files: srep:m3f:a2+exe2+lolz

Lolz Setting: -d128m -mt4 -mtt1 -lmdf1

Masquerade
23-09-2021, 04:37
Carldric Clement
It's just a unity game, xtool isn't needed. Nothing for it to precompress.

L33THAK0R
24-09-2021, 21:10
Does anyone know what "Oodle" algorithims "Planet Zoo" uses? I'm fairly sure it uses "Kraken" (output size was inflated a considerable amount when testing with the "Kraken" xtool plugin) but I'm unsure if additional algorithims are used. The scanner tool thingamajig outputs a "0" for all files (unknown/invalid).

mr_gnar
29-09-2021, 18:19
NEO: The World Ends With You
srep+lzma (Without staff roll videos)

Output size: 11.1 GB

8+ GB worth of USMs PLUS encrypted files...not worth messing with this unless you plan to re-encode them, and I don't even know if this one of those games where it has issues with touched USMs.

HunterEmrah
01-10-2021, 09:41
can someone please share actual method for Total War Warhammer II with rebuilding pack files?

dixen
01-10-2021, 10:06
can someone please share actual method for Total War Warhammer II with rebuilding pack files?No actual. with rebiulded pack new version TTWH2 - don't work

KaktoR
02-10-2021, 05:08
Total War: Warhammer II
v1.12.0 incl all DLC

10:57:36 - Selected ARC/DS method for Data1a-01.bin was: xzlib+srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
13:52:22 - Overall input size: 64.01 GB
13:52:22 - Overall output size: 38.75 GB (Ratio 60.54%)
13:52:22 - Data files creation time: 02:30:41

xtool ratio is about 106%. Not much, but that's in general in TW games.

movies contain vp8 files, sound wem.

panker1992
02-10-2021, 11:13
Total War: Warhammer II
v1.12.0 incl all DLC

10:57:36 - Selected ARC/DS method for Data1a-01.bin was: xzlib+srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
13:52:22 - Overall input size: 64.01 GB
13:52:22 - Overall output size: 38.75 GB (Ratio 60.54%)
13:52:22 - Data files creation time: 02:30:41

xtool ratio is about 106%. Not much, but that's in general in TW games.

movies contain vp8 files, sound wem.

Your method is incomplete, this game uses LZMA compression by their devs
you can REMOVE it with RPFM.

KaktoR
02-10-2021, 11:55
Your method is incomplete, this game uses LZMA compression by their devs
you can REMOVE it with RPFM.

Yes, but it would lead to bad crc and I don't know if the game works fine with not matching original crc.

dixen
02-10-2021, 23:53
Your method is incomplete, this game uses LZMA compression by their devs
you can REMOVE it with RPFM.On rebuilded terrain.pack - crashes Battle mode.

macut18
03-10-2021, 04:39
Them and Us
uelr+srep+4x4 = 25gb >> 7gb
- bpk on bik videos

@Thank KaktoR

KaktoR
03-10-2021, 05:14
Use uelr on *.tfc|*.u|*.udk|*.upk files

You can use bpk on bik videos.

KaktoR
03-10-2021, 05:28
Anno 1800
v12.1, all DLC, nothing ripped

17:00:46 - Selected ARC/DS method for Data1a-01.bin was: srep_new+razor_mtx
17:00:46 - Selected ARC/DS method for Data1b-01.bin was: xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4: fb273:lc8:mc1000000
--------------------------------------------------------------------------------------
21:19:20 - Overall input size: 68.95 GB
21:19:20 - Overall output size: 28.63 GB (Ratio 41.52%)
21:19:20 - Data files creation time: 03:59:40

hydefromt70s
04-10-2021, 14:22
uhhhh ok
2. Why are you using the cyberpunk 2077 precompressor on not cyberpunk 2077?????? :confused::confused::confused:

3. xzlib - we're getting somewhere, but this game is made in unreal engine 3 and is compressed with lzo and not zlib (zlib is ue4) - which is why you need to use UELR [Unreal Engine LZO Recompressor]

I know you write this post long time ago but just to clarify something. Cyberpunk 2077 wasn't made in UE3 as you said, it was made in CDProject's REDengine 4.

L33THAK0R
04-10-2021, 17:17
I know you write this post long time ago but just to clarify something. Cyberpunk 2077 wasn't made in UE3 as you said, it was made in CDProject's REDengine 4.

The title in question was "Rocket League" though, which was made in "Unreal Engine 3" and not "CDProjekts REDengine 4".

KaktoR
06-10-2021, 01:03
Do not use bpk on Alan Wake Remastered. Pretty much all videos give bad crc on extracting. I don't know if bpk has problems with 4K resolution or codec they use.

neko
06-10-2021, 03:26
Do not use bpk on Alan Wake Remastered. Pretty much all videos give bad crc on extracting. I don't know if bpk has problems with 4K resolution or codec they use.

Thanks for clarification. I have been confused and trying to figure what went wrong for the whole day :(

kj911
08-10-2021, 07:58
Noob questions: This ECM prefilter tested from very old/older game files (like: textures, audio, any formats) compressions? Use before any precompressors or before srep+lzma/lolz compressions? Yes, the filter optimized this CD-image files compression.

https://encode.su/threads/2829-RAZOR-strong-LZ-based-archiver?p=54307&viewfull=1#post54307
https://encode.su/threads/2829-RAZOR-strong-LZ-based-archiver?p=54336&viewfull=1#post54336

st0rm
08-10-2021, 09:13
Game Version : 1.3

Ripped - Re-encoded Files : Nothing

Original Size : 978 MB

Compressed Size : 748 MB

Install Time (Depends your System): 2-3 Minutes

Method : pzlib3+srep:m3f+lzma

Note(s) : Used Pzlib because game is older and low specs friendly. :cool:

FitGirl
09-10-2021, 09:36
Noob questions: This ECM prefilter tested from very old/older game files (like: textures, audio, any formats) compressions? Use before any precompressors or before srep+lzma/lolz compressions? Yes, the filter optimized this CD-image files compression.

https://encode.su/threads/2829-RAZOR-strong-LZ-based-archiver?p=54307&viewfull=1#post54307
https://encode.su/threads/2829-RAZOR-strong-LZ-based-archiver?p=54336&viewfull=1#post54336
ECM is a special filter, which removes/restores special ECC/CRC codes from CD ISOS. It's useless on anything else but ISOS with ECC codes.

:( Sad8669
12-10-2021, 10:27
Farcry 6 - Precompression

Library used : LZ4 v1.8.2 (Thanks to KaKtor)
Similar results can be obtained from LZ4 that comes with XTool v0.3.9.
XTool Version : 0.3.9
Database : Created using Razor's db (fc6.xtl)

[External compressor:xtool]
header = 0
packcmd = xtool.exe precomp -mlz4+fc6 -c32mb -t100p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

Compressed 1 file, 683,722,320 => 1,286,884,486 bytes. Ratio 188.22%
Compression time: cpu 1.02 sec/real 48.29 sec = 2%. Speed 14.16 mB/s
All OK

lomesmo
16-10-2021, 11:51
A Hat In Time - Ultimate Edition

All DLC
Input size: 8.37 GB
Output size: 3.88 GB
Method: xLZO+srep+exe+7z+lzma2

lomesmo
16-10-2021, 11:53
AKIBA'S TRIP - Hellbound & Debriefed

Input size: 2.47 GB
Output size: 1.33 GB
Method: xZLib+srep+7z+lzma2

lomesmo
16-10-2021, 13:48
Bullet Witch

All DLC

Input size: 4.57 GB
Output size: 3.09 GB
Method: xZLib+srep+lzma2

L33THAK0R
16-10-2021, 18:28
Hi all,

Has anyone given "Call of Duty: Black Ops III (3)" a shot? It piqued my interest when I saw the QuickBMS script for "xpack" archives (https://imgur.com/a/O5HQHss)[1] (https://aluigi.altervista.org/quickbms.htm) mentioned that the "LZO", "LZ4" & "Oodle" compression algorithms were used within these archives. I'm sure testing which libraries for "LZ4" give the best output won't be too tricky, but in regards to the "Oodle" codec(s) used, I'm a bit stumped. As far as I know a "oo2core_#_win64.dll" is required, with either "xtool" or "Oo2Rec#", to pre-compress a given file, however such a file isn't present within the root and sub-directories of the title in question. Does anyone have any idea how to go about pre-compressing the "Oodle" streams for this title? Apologies if I buggered the terminology.

KaktoR
17-10-2021, 03:27
Well you can check with hex edit if the game has oodle at all (I don't know), if yes then just try all different libraries.

kj911
17-10-2021, 14:32
WAVE compression again: Retested few switches, use TinyGamePack tools by Panker1992 and from searching best results.

In last time good results: 635 350kB
....



Newest results, from previous months testing:

MSC+SREP: 634 861kB (Use TAK 2.3.0)
MSC+SREP: 634 836kB (Use TAK 2.3.1)

Retested and again use MSC with TAK 2.3.1 and use with any extra switches and goal!!

New Results: 630 595kB!! Nice, few MB's gain from previous versions. Overmaxed WAV compreesion. :)

Tiny SFX compressor commands:
arc a -lc1024 -ld1024 -ep1 -ed -r -w.\ data.bin -mmsc+rep:256mb+4x4 "Data\*"

arc.ini:
[External compressor:msc]
header = 0
packcmd = MSC c -v -f -wav=1 -raw=2 -bmp=1 -ddsraw=1 -ddsdxt=1 -mp3=2 -bmf=9s -tak=9 -dxt=2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

CRCs are OK! My preference, smallest memory eating, (lower than 768MB/1GB) from during decompress, heavy compressed, repacked games to use old PC's. (Win XP with min. 1-1.25GB RAM or Win7 with 2GB RAM memory.)

More screens:

https://i.kek.sh/MaGjls4ga1X.jpg
https://i.kek.sh/468nxWxsqpv.jpg
https://i.kek.sh/JK1wgOZUDQF.jpg
https://i.kek.sh/9kw96quv88w.jpg

KaktoR
21-10-2021, 02:41
NieR Replicant ver 1 22474487139

Anyone knows what zstd lib to use? It's slow af

Masquerade
21-10-2021, 06:26
NieR Replicant ver 1 22474487139

Anyone knows what zstd lib to use? It's slow af

Unsure, I have checked with latest ZSTD to see if it can list info about the archives, yet it can only seem to load info for few archives. Yet, all files are definitely compressed with zstd.

for %f in (*.arc) do zstd -l "%f"
Frames Skips Compressed Uncompressed Ratio Check Filename
1 0 9.25 MB 30.19 MB 3.263 None common.arc
Frames Skips Compressed Uncompressed Ratio Check Filename
1 0 690.86 KB 1748.36 KB 2.531 None info.arc
Frames Skips Compressed Uncompressed Ratio Check Filename
File "init.arc" not compressed by zstd
Frames Skips Compressed Uncompressed Ratio Check Filename
File "lang1.arc" not compressed by zstd
Frames Skips Compressed Uncompressed Ratio Check Filename
File "lang2.arc" not compressed by zstd
Frames Skips Compressed Uncompressed Ratio Check Filename
File "param.arc" not compressed by zstd
Frames Skips Compressed Uncompressed Ratio Check Filename
File "stream.arc" not compressed by zstd

I even tried compressing a demo file with zstd and testing the latest libzstd to see if the precompressor worked, neither zstdrec or xtool could precompress a random file compressed with latest zstd.

KaktoR
21-10-2021, 06:46
Yes, the first file contains just 1 block with zstd (28B52FFD).

However, ratio with lzma2 for init.arc is about 70%.

I will investigate it further :D

Masquerade
23-10-2021, 02:53
KaktoR
You can use the following tools to extract all of the compressed ZSTD streams from NieR replicant. It will allow for decompression of 99% of streams.

1. VGM Toolbox (https://sourceforge.net/projects/vgmtoolbox/) and this script (https://bayfiles.com/f6t6M2Pfu7/nierreplicant_xml). Download latest zstd.exe.

2. Place the XML file in plugins\AdvancedCutter folder of VGMToolbox.

3. Open VGMToolbox, go to Misc. Tools > Extraction Tools > Generic > Advanced Cutter

4. From the drop down menu in the top right, select NieRReplicant and click LOAD.

5. Next, you can drop each of the arc files into the top left box and it will extract the streams. You only really need to do this for the large arc files. No need on the little ones.

6. After extracting the streams, run this python script (https://bayfiles.com/nauaM8Peu4/trimmer_py) for each of the files. This will remove the extra padding bytes (@) that the devs added so that each stream begins at an offset of divisible by 16. ZSTD will not decompress files that contain data after the end of the stream.

for %f in (*.MSQ) do trimmer.py "%f"

Ignore that it involves .MSQ extension. You can switch it to .TMP or whatever.

7. for /r %f in (*.MSQ) do zstd -d "%f" -o "%~df%~pf%~nf.DEC" This will use zstd to decompress the streams. Afair it fails on about 16 streams out of ~7800, but you can decompress most of the game this way.

KaktoR
23-10-2021, 06:10
Question is if it's profitable because with plain srep+lzma i get 13.5GB

Masquerade
23-10-2021, 11:25
Question is if it's profitable because with plain srep+lzma i get 13.5GB

Oh, most certainly not profitable in the slightest. But still good fun! :D

hwang4
26-10-2021, 01:30
Hi!Need A valid unreal key for "Chinese Paladin - Sword & Fairy 7"! :)
Should i use "ue4dt" or xtool with unreal plugin?thanx!

KaktoR
26-10-2021, 01:47
0xE53BEBDBCEC82995F37033B62CFD8243E02C10C03AB12EA6 82F654DC805F4009
In my tests there were no difference in using ue4dt or xtool.

dixen
26-10-2021, 01:52
Keys
Sword and Fairy 7 (cubejoy) 0xB37742E2DF203FAF02D00DD495D3297FAEB4B14D89FBD6C8 8D874884FAB8EAD5
Sword and Fairy 7 (steam) 0xE53BEBDBCEC82995F37033B62CFD8243E02C10C03AB12EA6 82F654DC805F4009

hwang4
26-10-2021, 07:17
KaktoR , you are right!I tried with both ue4dt & xtool without success!
So ? What's should i use for better compression ?

KaktoR
26-10-2021, 07:21
I think the files are not compressed anyway, but I have not checked further.

KaktoR
26-10-2021, 07:31
NASCAR 21 Ignition

15:00:20 - Selected ARC/DS method for Data1a-01.bin was: xtool:mpreflate+srep_new+4x4:b64mb:lzma:ultra:64m: bt4:fb273:lc8:mc1000000
15:00:20 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
16:25:17 - Overall input size: 18.62 GB
16:25:17 - Overall output size: 12.12 GB (Ratio 65.07%)
16:25:17 - Overall conversion time: 01:24:52

Data1a *.pak
Data1b rest of the files

Masquerade
26-10-2021, 10:02
Decimal locations for unneeded voiceovers in The Dark Pictures Anthology: House of Ashes

pakchunk0-WindowsNoEditor.pak

DE - 7609391157 - 7855465402

ES - 7855466549 - 8101614621

FR - 8101615669 - 8350358791

IT - 8350359605 - 8589641034

RU - 8589641781 - 8828443389

KaktoR
26-10-2021, 10:50
Decimal locations for unneeded voiceovers in The Dark Pictures Anthology: House of Ashes

pakchunk0-WindowsNoEditor.pak

DE - 7609391157 - 7855465402

ES - 7855466549 - 8101614621

FR - 8101615669 - 8350358791

IT - 8350359605 - 8589641034

RU - 8589641781 - 8828443389

Thanks. Also do not use bpk on this. Will cause crc errors (again) :D

KaktoR
28-10-2021, 00:53
Marvel's Guardians of the Galaxy

23:55:06 - Selected ARC/DS method for Data1a-01.bin was: srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
01:50:56 - Overall input size: 75.94 GB
01:50:56 - Overall output size: 23.50 GB (Ratio 30.95%)
01:50:56 - Overall conversion time: 01:53:38

Masquerade
28-10-2021, 08:57
Decimal locations for Japanese voiceover in SCARLET NEXUS v1.04:

pakchunk7-WindowsNoEditor.pak

0 - 353837341
Remaining JP voices are inside pakchunk70-WindowsNoEditor.pak.

Total: 1.25GB

KaktoR
28-10-2021, 09:53
SCARLET NEXUS
v1.04

17:28:59 - Selected ARC/DS method for Data1a-01.bin was: xtool:c32mb:d1:mreflate:mscarletnexus+srep_new+4x4 :b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
18:52:54 - Overall input size: 17.49 GB
18:52:54 - Overall output size: 13.00 GB (Ratio 74.36%)
18:52:54 - Overall conversion time: 01:23:52

KaktoR
28-10-2021, 10:48
FATAL FRAME / PROJECT ZERO Maiden of Black Water

18:54:44 - Selected ARC/DS method for Data1a-01.bin was: srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
19:47:21 - Overall input size: 22.68 GB
19:47:21 - Overall output size: 11.51 GB (Ratio 50.77%)
19:47:21 - Overall conversion time: 00:52:35

Masquerade
28-10-2021, 12:17
FATAL FRAME / PROJECT ZERO Maiden of Black Water

18:54:44 - Selected ARC/DS method for Data1a-01.bin was: srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
19:47:21 - Overall input size: 22.68 GB
19:47:21 - Overall output size: 11.51 GB (Ratio 50.77%)
19:47:21 - Overall conversion time: 00:52:35

With lolz, 11.1GB.

hwang4
28-10-2021, 15:04
Chinese Paladin - Sword & Fairy 7

Need Some Help! when using ue4dt with "m1" parameter instead of "m2" , i got better compression but with crc check error !is there any solution to fix this ? i tried xtool with unreal plugin , but the "ue4dt with m1" remains the better method!

KaktoR
30-10-2021, 01:58
Anyone can tell me something about Age of Empires 4? I don't want to download 200 GB just for testing (I don't wanna play this game anyway) :)

dixen
30-10-2021, 03:34
Anyone can tell me something about Age of Empires 4? I don't want to download 200 GB just for testing (I don't wanna play this game anyway) :)

Age of Empires IV (4) [P] [RUS + ENG + 12 / ENG] (2021) (5.0.7274.0) [UWP-Dump] 34 gb

DX12AssetsChinese.sga

Compressed 1 file, 940,809,153 => 1,435,395,740 bytes. Ratio 152.57%
Compression time: cpu 2.03 sec/real 105.07 sec = 2%. Speed 8.95 mB/s
All OK
zlib

KaktoR
30-10-2021, 13:35
Age of Empires 4
1 language, without HD/4K files

21:00:50 - Selected ARC/DS method for Data1a-01.bin was: xtool:mreflate+srep_new+4x4:b64mb:lzma:ultra:64m:b t4:fb273:lc8:mc1000000
21:00:50 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
21 22:34:10 - Overall input size: 35.08 GB
22:34:10 - Overall output size: 21.96 GB (Ratio 62.61%)
22:34:10 - Overall conversion time: 01:33:14

Data1a = *.sga
Data1b = rest of the files

kj911
31-10-2021, 04:35
Repacked again my very old MSTS REPACK compare the newest compression techniques.

MSTS 2004 HUN (repacked in year 2010!* Newest repack in last time 2021):

Old, ace*pcf.arc, 6 files: 1 694 461kB (ARC file, store --> PCF file --> ARC -mx)
New, one Data-1.arc file: 998 000kB (xZLib+srep+lolz)
Compare1: 1 742 537kB (no zlib precompression, SREP+LZMA only)
Compare2: 1 759 557 kB (ARC -mx only)
Compare3: 1 147 153 kB (xZLib+srep+LZMA)

Old, Data_arc.arc: 465 445kB (ARC file, rep:64mb --> ARC -mx)
New, Data-2.arc: 401 611kB (xZLib+srep+lolz)

Old, Snd.uha file: 186 970kB (ARC with TTA --> Uharc 0.6 -m3)
New, Snd-0.arc file: 181 873kB (msc+rep+4x4)

Binaries, old: 4225kB --> New: 4102kB (ARC only compression.)**

*use with P4 and 768MB RAM!
Yes! ~2.245GB ---> ~1.515GB. Installed game size: 7.4GB
Old version install time with Pentium B430 CPU and 1GB RAM: 62 minutes. (+/- few seconds.)
New version install time with Pentium B430 CPU and 1GB RAM: ~45 minutes! (44m38.55s) :cool:

**arc.exe a -lc1024 -ld1024 -ep1 -ed -s -r -w.\ Data.bin -mrep:256mb+exe2+delta+lzma:128mb:ultra:bt4:fb273:m c1023:lc8 "Data\*"

Note: xZLIB 0.12 x86, SREP x64 with m3f:a2 and LOLZ defaults with modified to d128m:tt16

See: https://i.kek.sh/aPm68PqdBfC.png

Any huge compression results attached in the file. Its will compress now ~22GB's MSTS game. :D (3nd variant with mods.)

Masquerade
04-11-2021, 00:15
Step 1: Decrypt the game files:


Download the decryption tool from the attachment below (credits to atom0s & Gabriel Dehan). This EXE is specific to game version v1.1.4.6.
Drop the EXE into the game's root folder.
Open CMD prompt here.
Run the following command:
for /r %f in (*.pak) do Pak_Decrypt "%f" "%~df%~pf%~nf.zip"
Once the above command completes, run the next command:
for /r %f in (*.zip) do del "%~df%~pf%~nf.pak"
Once the above command completes, run the next command:
for /r %f in (*.zip) do ren "%f" "%~nf.pak"

Now you have a fully decrypted game.

Step 2: Compression


Make 2 archives.
Archive 2 will contain "Sounds_MISC.pak" and "Sounds_WEM.pak".
Archive 1 will contain everything else.


Start Size: 29.6GB

Method for archive 1:
xtool-zlib+srep+lolz
Method for archive 2:
srep+lzma

Final Compressed Size: 12.1GB

Masquerade
04-11-2021, 13:36
Injustice 2 Precompression Test

Using oo2core_4_win64.dll from game's Binaries folder.

XTool v0.3.21 (kraken codec):
Compressing 1 file, 100,237,312 bytes
Compressing BGND_RedSunPrison_ScriptAssets.xxx
Compressed 1 file, 100,237,312 => 103,442,017 bytes. Ratio 103.20%
Compression time: cpu 0.19 sec/real 78.34 sec = 0%. Speed 1.28 mB/s
All OK

oo2reck:
Compressing 1 file, 100,237,312 bytes
Compressing BGND_RedSunPrison_ScriptAssets.xxx
Compressed 1 file, 100,237,312 => 183,527,393 bytes. Ratio 183.09%
Compression time: cpu 0.08 sec/real 33.68 sec = 0%. Speed 2.98 mB/s
All OK

KaktoR
04-11-2021, 14:51
Injustice 2 Precompression Test

Using oo2core_4_win64.dll from game's Binaries folder.

XTool v0.3.21 (kraken codec):
Compressing 1 file, 100,237,312 bytes
Compressing BGND_RedSunPrison_ScriptAssets.xxx
Compressed 1 file, 100,237,312 => 103,442,017 bytes. Ratio 103.20%
Compression time: cpu 0.19 sec/real 78.34 sec = 0%. Speed 1.28 mB/s
All OK

oo2reck:
Compressing 1 file, 100,237,312 bytes
Compressing BGND_RedSunPrison_ScriptAssets.xxx
Compressed 1 file, 100,237,312 => 183,527,393 bytes. Ratio 183.09%
Compression time: cpu 0.08 sec/real 33.68 sec = 0%. Speed 2.98 mB/s
All OK

Remove languages from video files with ffmpeg (you have to rename scx to mp4)

Masquerade
05-11-2021, 00:23
INJUSTICE 2 Audio tracks inside videos:

https://i.ibb.co/8MgRVqx/Capture.png

kj911
05-11-2021, 02:07
lolz v21a7 > -dt1 -dtp1 -dtb1 -dto1 -dtm1 -dtw1 -dtd1 -mtt1 -mt8 -d128m -oh14 -fba4096 -mc1023 -bc8 -blr8 -bm8 -pc4

Better compression switches from LOLZ compression compare, than ones: -dt1 -dtp1 -dtb1 -dto1 -dtm1 -dtw1 -dtd1 -mtt1 -mt8 -d128m -fba4096 -mc1023 ???

This ttXX switches usage? Higher tt values = better compression rate or not? (tt16/32/64, etc.. Default: tt4/8) Or data type different? bc/blr/bm and pc switches, reasonable effect to better compression rate? No experience of this?

dixen
05-11-2021, 07:38
INJUSTICE 2 Audio tracks inside videos:

https://i.ibb.co/8MgRVqx/Capture.png

12,2 gb > 9.4 gb
*.xxx, *.tfc - Xtool+oo2core plugin

KaktoR
05-11-2021, 09:03
Any info about Miner's Mettle?

Masquerade
05-11-2021, 12:12
Any info about Miner's Mettle?

Wasn't able to get srep 3.92 beta working with it, both in $$arc$$ or stdio.

KaktoR
05-11-2021, 12:30
3.93a works, but I don't know anything yet about precompression

KaktoR
06-11-2021, 09:27
Call of Duty: Vanguard
oo2core_8_win64.dll from game folder


oo2reck
Compressed 59 files, 62,292,776,931 => 80,613,566,269 bytes. Ratio 129.41%
Compression time: cpu 54.55 sec/real 7302.48 sec = 1%. Speed 8.53 mB/s
All OK

xtool:kraken
Compressed 59 files, 62,292,776,931 => 85,178,762,171 bytes. Ratio 136.74%
Compression time: cpu 53.41 sec/real 6224.33 sec = 1%. Speed 10.01 mB/s
All OK

First time I see xtool:kraken is better then oo2rec

KaktoR
06-11-2021, 13:16
Forza Horizon 5

*.zip, *.minizip -> zlib
do not use bpk on bk2 files, same shit as always (crc error). I guess bpk has reached its limits.

:( Sad8669
06-11-2021, 21:13
KaktoR,

Actually, it isn't the first time xtool:kraken is yielding better results than XTool.

Take a look at this, it is a sample from WWE 2K19.

XTool - Kraken
Compressed 1 file, 3,291,136 => 21,504,257 bytes. Ratio 653.40%
Compression time: cpu 0.00 sec/real 15.72 sec = 0%. Speed 0.21 mB/s
All OK

OodleRec - Kraken
Compressed 1 file, 3,291,136 => 5,251,731 bytes. Ratio 159.57%
Compression time: cpu 0.00 sec/real 4.97 sec = 0%. Speed 0.66 mB/s
All OK[/QUOTE]

you are kinda right about bpk, had same issues when i tried it on WDL bpks.

I guess encoding bpks is the only option we have, if the game accepts the encoded ones ofcourse.

KaktoR
07-11-2021, 05:46
Miner's Mettle

12:23:16 - Selected ARC/DS method for Data1a-01.bin was: srep+LZMA-MT
----------------------------------------------------------------------------------------------------------------------------------------
14:45:47 - Overall input size: 48.16 GB
14:45:47 - Overall output size: 17.25 GB (Ratio 35.82%)
14:45:47 - Overall conversion time: 02:22:26

Masquerade
10-11-2021, 05:42
ENGLISH ONLY! All unneeded language files + deluxe movies ripped.

Start size: 51.3GB

Compression method:
oo2recm+xtool_zlib+srep+lolz

Do not use XTool chunk size of greater than 128mb otherwise it will get stuck during compression. (tested on my 64GB, 16 thread PC). XTool v0.12 used. Thanks to Razor12911 for the tools.

Final Size: 37.9GB

KaktoR
10-11-2021, 06:20
@Masquerade
Is it really worth it? :D

https://i.imgur.com/WKKK12U.png

panker1992
10-11-2021, 08:16
it seems that you 2 are crancking it up :D

glad to see active people, BTW my Final size on AoE - IV

8GB

Masquerade
10-11-2021, 09:37
@Masquerade
Is it really worth it? :D
IMO yes, considering there's little to no speed penalty.

I didn't test selkie though, perhaps you can get even more inflation with that.

KaktoR
11-11-2021, 10:08
Why the new GTA trillogy has oodle files in place when they have zlib? :confused:

KaktoR
12-11-2021, 01:24
Grand Theft Auto III Definitive Edition

10:04:11 - Selected ARC/DS method for Data1a-01.bin was: xt_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb27 3:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
10:23:20 - Overall input size: 4.56 GB
10:23:20 - Overall output size: 3.36 GB (Ratio 73.82%)
10:23:20 - Overall conversion time: 00:19:04

KaktoR
12-11-2021, 02:49
Grand Theft Auto San Andreas Definitive Edition

10:24:52 - Selected ARC/DS method for Data1a-01.bin was: xt_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb27 3:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
11:49:21 - Overall input size: 19.75 GB
11:49:21 - Overall output size: 14.87 GB (Ratio 75.31%)
11:49:21 - Overall conversion time: 01:24:26

KaktoR
12-11-2021, 03:34
Grand Theft Auto Vice City Definitive Edition

11:50:40 - Selected ARC/DS method for Data1a-01.bin was: xt_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb27 3:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
12:33:58 - Overall input size: 9.68 GB
12:33:58 - Overall output size: 7.24 GB (Ratio 74.76%)
12:33:58 - Overall conversion time: 00:43:14

KaktoR
12-11-2021, 10:06
ENGLISH ONLY! All unneeded language files + deluxe movies ripped.

Start size: 51.3GB

Compression method:
oo2recm+xtool_zlib+srep+lolz

Do not use XTool chunk size of greater than 128mb otherwise it will get stuck during compression. (tested on my 64GB, 16 thread PC). XTool v0.12 used. Thanks to Razor12911 for the tools.

Final Size: 37.9GB

13:03:18 - Selected ARC/DS method for Data1a-01.bin was: oo2recm8+xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra :64m:bt4:fb273:lc8:mc1000000
13:03:18 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
19:01:31 - Overall input size: 51.51 GB
19:01:31 - Overall output size: 44.30 GB (Ratio 86.01%)
19:01:31 - Data files creation time: 05:31:39

Just 200mb for lolz, bad.

Even worse that with xtool applied the size is bigger then my last repack with just oo2recm https://fileforums.com/showpost.php?p=490460&postcount=2136

KaktoR
12-11-2021, 11:32
They are Billions
v1.1.4

19:48:52 - Selected ARC/SPLIT method for Data1a-01.bin was: srep_new+lolz:d64:mtt1:mt12:mc1023/$storepack=FA-MIN/$libpack=srep_new+razor_mtx/$oggretheyarebillions=oggre/$deflatetheyarebillions=xt_zlib+srep_new+lolz:d64: mtt1:mt12:mc1023
----------------------------------------------------------------------------------------------------------------------------------------
20:30:49 - Overall input size: 5.16 GB
20:30:49 - Overall output size: 1.84 GB (Ratio 35.70%)
20:30:49 - Overall conversion time: 00:41:53

KaktoR
12-11-2021, 11:59
Battlefield 2042 test

srep+4x4:lzma
Overall input size: 246.54 MB
Overall output size: 55.90 MB (Ratio 22.67%)

oo2reck+srep+4x4:lzma (oo2reck takes very long)
Overall input size: 246.54 MB
Overall output size: 54.91 MB (Ratio 22.27%)

xtool:kraken+srep+4x4:lzma
Overall input size: 246.54 MB
Overall output size: 54.94 MB (Ratio 22.28%)

KaktoR
12-11-2021, 15:39
Skyrim Anniversary Edition test

Compressed 15 files, 9,955,981,643 => 16,269,399,549 bytes. Ratio 163.41%
Compression time: cpu 11.56 sec/real 320.70 sec = 4%. Speed 31.05 mB/s
All OK

Extracted 15 files, 16,269,399,549 => 9,955,981,643 bytes. Ratio 163.41%
Extraction time: cpu 17.92 sec/real 271.19 sec = 7%. Speed 36.71 mB/s
All OK

Data\Skyrim - Animations.bsa
Data\Skyrim - Interface.bsa
Data\Skyrim - Meshes0.bsa
Data\Skyrim - Meshes1.bsa
Data\Skyrim - Misc.bsa
Data\Skyrim - Shaders.bsa
Data\Skyrim - Textures0.bsa
Data\Skyrim - Textures1.bsa
Data\Skyrim - Textures2.bsa
Data\Skyrim - Textures3.bsa
Data\Skyrim - Textures4.bsa
Data\Skyrim - Textures5.bsa
Data\Skyrim - Textures6.bsa
Data\Skyrim - Textures7.bsa
Data\Skyrim - Textures8.bsa



Working bsa plugin https://fileforums.com/showpost.php?p=494532&postcount=223
Use lz4 v1.8.3 library

PS: t100p stuck for me, so use a lower thread usage for xtool

kj911
14-11-2021, 01:13
Microsoft Train Simulator *.ACE files:

older: xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt32:fba4096:mc1023

newest: xZLib+srep:m3f:a2+lolz:dt:dtb1:dtw1:dtm1:dto1:dtd1 :mtt1:mt4:d128m:tt1:oh14:gm01:lm4:fba4096:mc1023

Green its optional switches, probably testing any game. Defaults, its 00, 11 or 21 worse results than 00.
Red the magic switch! Nice compression gain! And overall usage all physical RAM! (8GB and 87% usage vs. 97-99%)
oh14 its minimally better compression. Defaults: oh12
tt1 its very fast mode, max usage: tt16 or tt32!! Higher values not better and extremely slow compression.

Difference? Old version 2.93GB! New, with tt1 (compress again with tt16 and d160m switches in last time)* 2.80GB!! ~5% compression rate gain.

UPDATE: Aaarrgh! The test package don't decompresseable! CLS-ini error or any tips?

*tt32 its probably very best results. Calculated compression time avg. ~2days! Overkill.

Cesar82
16-11-2021, 06:19
Could someone familiar with LOLZ parameters help me with some questions?

How is actual memory usage calculated using for multi thread used An X value for dictionary (-d)?

Considering that the supported values for dictionary are from 0 to 2047 "-d[0..2047]): 64m". If you enter a value above 2047 (such as 2050) will the default value be set to the dictionary (64m) or will the maximum value (2047m) be used?


I need this information because I created presets for LOLZ in DiskSpan GUI.
The memory and thread information will be obtained when starting the DiskSpan GUI and then starting the compression will be calculated based on the threads and RAM of the system.
It will also be informed the number of system theads replaced <thread> by the number of virtual cores (threads) of the current CPU during the conversion.
So one more question arises...
In the lolz documentation it states that the supported values are 1 to 16 thread: "mt threads(-mt[1..16]): 1"
If using a Ryzen processor 9, 5900x is 24 threads, so if you use -mt24, lolz will set it to 1 (default) or to 16 (Maximum supported).

Below is the example where it will be in LOLZ presets in DiskSpan GUI.
LOLZ_HIGH = lolz:dtb1:d<expr:0.65*memory/(thread+1)/1048576>m:mtt1:mt<thread>:mc1023

Expression example: <expr:0.65*memory/(thread+1)/1048576>

0.65 = percentage of calcule (65%)
memory = full system memory (RAM)
thread = number of logical cores
core = number of Physical cores
1048576 = convert result in bytes to mb

If someone can help me with these questions, I would appreciate it!

kj911
16-11-2021, 08:47
Its calculated result correct?? See the linked picture from my friend laptop specifications. --> https://i.kek.sh/zUjUwL6MVZE.jpg

Example: Intel Core I3-1005G1 CPU, Win10 x64 with 8GB memory. (Total cores: 2, Total Threads: 4)

Available physical RAM 8GB --> 8589934592 byte

PreCalculate: 0.65*8589934592 = 5583457484,8
Cores --> 4+1 = 5
Final calculate: 5583457484,8/5/1048576 = 1064.96 MB?? Try it added extra /8 values? 132MB values its really.

Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

Cesar82
16-11-2021, 15:35
Use -mtt1 and -mt4 mode and -d128m dictionary size, eats 5.2-5.5GB's memory from during compression.

For your system it would result as if using from the command line:
LOLZ_HIGH = lolz:dtb1:d1065m:mtt1:mt4:mc1023

So lolz would use: 4 * 1065 = 4260 MB of 8192 MB.

I believe the memory usage of lolz is calculated as: theads*dictionary

Example: If using a Ryzen 5900x (24 theads) and 32 GB of RAM we would have:
LOLZ_HIGH = lolz:dtb1:d852m:mtt1:mt24:mc1023

So lolz would use: 24 * 852 = 20448 MB of 32768 MB.

It may be recommended to do the calculation using available memory rather than total system memory.

Maybe it's not just simply multiplying the dictionary by the number of threads, that's my question.
Can not understand. See the attached video.

its_gnar_again
16-11-2021, 18:22
Has anyone here had any luck getting Final Fantasy X/X-2 HD to decompress without any issues? The method on here is basically xtool(xzlib)+srep+lolz, but that ends up with a corrupted archive error during decompression not even half-way. I have tried the following:


Tried different configs for LOLZ
Tried different configs for SREP
Tried outright replacing LOLZ with LZMA
Tried various versions of XTool and ZTool, and different configs for them as well.


I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

KaktoR
17-11-2021, 09:39
Hellblade: Senua's Sacrifice Enhanced
Video documentary ripped

17:51:27 - Selected ARC/DS method for Data1a-01.bin was: srep_new+LZMA-MT
17:51:27 - Selected ARC/DS method for Data1b-01.bin was: bpk
----------------------------------------------------------------------------------------------------------------------------------------
18:38:43 - Overall input size: 17.25 GB
18:38:43 - Overall output size: 8.25 GB (Ratio 47.84%)
18:38:43 - Data files creation time: 00:38:17

KaktoR
17-11-2021, 11:48
Has anyone here had any luck getting Final Fantasy X/X-2 HD to decompress without any issues? The method on here is basically xtool(xzlib)+srep+lolz, but that ends up with a corrupted archive error during decompression not even half-way. I have tried the following:


Tried different configs for LOLZ
Tried different configs for SREP
Tried outright replacing LOLZ with LZMA
Tried various versions of XTool and ZTool, and different configs for them as well.


I really don't know what the problem is with this game, and trying to compress the 2 games separately still results in a corrupted error too during decompression.

xtool:reflate works just fine for me.

[External compressor:xtool]
header = 0
default = -c64mb -t75p
packcmd = "XTool.exe" precomp { -option} - - <stdin> <stdout>
unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

its_gnar_again
17-11-2021, 17:06
xtool:reflate works just fine for me.

[External compressor:xtool]
header = 0
default = -c64mb -t75p
packcmd = "XTool.exe" precomp { -option} - - <stdin> <stdout>
unpackcmd = "XTool.exe" decode -t100p - - <stdin> <stdout>

Did you test only xtool to see if it works?

My xtool setup is pretty much the same from when I last tested it.

[External compressor:xtool]
header = 0
packcmd = XT\xtool.exe precomp -mzlib -c64mb -t75p --dbase - - <stdin> <stdout>
unpackcmd = xtool.exe decode -t100p - - <stdin> <stdout>

I haven't tested JUST xtool only to see if that works, but I will get back to you here from when I do. I don't understand what the problem is on my part really.

Grumpy
17-11-2021, 22:22
@its_gnar_again

forgot the pass & email I used on my old account lmao

Surely you remember one of them, you've had 4 or 5 accounts since 2020 :eek:

its_gnar_again
18-11-2021, 19:05
I haven't tested JUST xtool only to see if that works, but I will get back to you.


UPDATE: I tried specifically using reflate on xtool+srep+lolz, and still get a decompression error. However, I did try using xtool only and it decompressed just fine...what could honestly be the issue? I would like to get this figured out. :confused:

KaktoR
19-11-2021, 00:32
Maybe check your CLS.ini settings for lolz and srep.

And make sure you're using the correct cls files for lolz and srep

Edison007
19-11-2021, 08:10
How is actual memory usage calculated using for multi thread used An X value for dictionary (-d)?
[dictionary*~11 + (mtb*2) + (memory_for_models)] * threads

frankiedylan
19-11-2021, 15:07
ZLib and LolZ are probably the best. I used them on my old projects back in the day and they performed well.

Cesar82
19-11-2021, 17:42
[dictionary*~11 + (mtb*2) + (memory_for_models)] * threads
Thanks for the info.
Don't understand about "memory_for_models"?
You used '"[' and "]", have the same meaning as parentheses for the calculation?
It depends on what parameters?

In these presets below what would the calculation be, give me an example so I can understand.
LOLZ_FAST = lolz:d32m:mtt1:mt<thread>:fba4096:mc2;
LOLZ_NORMAL = lolz:d64m:mtt1:mt<thread>:mc1023;
LOLZ_HIGH = lolz:dtb1:d128m:mtt1:mt<thread>:mc1023;
LOLZ_ULTRA = lolz:dtb1:d256m:mtt1:mt<thread>:mc1023

Thread is detected by DSG.
Consider as an example on a 24 thead cpu (I think lolz trunc to 16).

-----------------
This calculation is correct (This without adding the use "memory_for_models" which I didn't understand)?
To CPU 5900x (24 threads) lolz trunc to 16 threads
lolz default mtb is 32m
lolz:dtb1:d256m:mtt1:mt16:mc1023;
calc: (256m*11 + (32m*2) + (?)) * 16
(2816m + 64m + ?) * 16
2880m * 16 = 46080m

Edison007
20-11-2021, 05:08
You used '"[' and "]", have the same meaning as parentheses for the calculation?
Yes, priority of math operations)

calc: (256m*11 + (32m*2) + (?)) * 16
(2816m + 64m + ?) * 16
2880m * 16 = 46080m
Yep, but the block size will be - 256 mb.

Quotes translated with Google Translate.
-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, so the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;
lolz test19g
- in the -mtt1 mode, when compressing, the block and dictionary sizes are aligned with each other according to the larger of them;

I don’t remember the required memory for the models. But this is a trifle

Cesar82
20-11-2021, 07:13
Thank you very much Edison007.
-mtb [2..512] - sets the block size in MB. When -mt1 -mtt0 also plays a role, but minimal. And more does not mean better. Usually, for -mtt0, the optimal value is about 32-64MB, so the size of the dictionary should be more than 2 times larger. For mtt1 mode, the dictionary size must be no more than the block size;
I think the problem was this. I was using default value for block size and was using value greater than 512 for dictionary using mtt1.

PS: I noticed that if you compress as small as 300MB the value set for dictionary in console output during compression is smaller than if you compress as much as 30GB. Dictionary appears to be input size dependent.

I will run some tests and see the results.
Thanks!

Edison007
20-11-2021, 07:50
PS: I noticed that if you compress as small as 300MB the value set for dictionary in console output during compression is smaller than if you compress as much as 30GB. Dictionary appears to be input size dependent.

Another quote from changelog:
lolz test19c
- added reduction of the dictionary and the block to an acceptable size if the input file is smaller than the sizes of these parameters;

kj911
21-11-2021, 10:09
MSC Questions: Use frog=9 switches equal to --bestnew or any OFR switches??

Example: chimes.wav file in 216 364 byte

MSC with frog=9 switches: 36 455 byte (internally from ARC archive: 36 233 byte)
Directly packing OFR with use --maximumcompression switches: 35 608 byte

Edison007
21-11-2021, 11:09
kj911
MSC version 0064 (ofr 4.910b)
ArrOfFROGParams:array [1..9]of PAnsiChar=('
--mode fast --experimental --seek min --optimize best',
'--mode normal --experimental --seek min --optimize best',
'--mode high --experimental --seek min --optimize best',
'--mode extra --experimental --seek min --optimize best',
'--mode best --experimental --seek min --optimize best',
'--mode highnew --experimental --seek min --optimize best',
'--mode extranew --experimental --seek min --optimize best',
'--mode bestnew --seek min --optimize best',
'--maximumcompression --experimental --seek min --optimize best');

+

https://krinkels.org/threads/msc-media-streams-compressor.1587/page-7#post-33503

dixen
22-11-2021, 05:49
What about *.shapes from Farming Sim 2022? Encrypt files?

Masquerade
22-11-2021, 10:47
What about *.shapes from Farming Sim 2022? Encrypt files?

Likely: https://zenhax.com/viewtopic.php?t=4030

dixen
27-11-2021, 11:45
Hi, my friends)
What about decrypt/decompress *.FF from CoD BO 2?)

KaktoR
04-12-2021, 02:09
Chorus
Do not use binkpack on video files. Again bpk fails because of 4k resolution, used codec or something else. RIP

Masquerade
04-12-2021, 08:48
Grab oo2core from game files and rename from oo2core_8_win64.dll to oo2core_7_win64.dll.

oo2reck+srep+lolz

4.2GB ----> 2.9GB

Masquerade
05-12-2021, 01:03
Grab oo2core from game files and rename from oo2core_8_win64.dll to oo2core_7_win64.dll.

oo2reck+srep+lolz/$mp4=srep

8.7GB ----> 6.2GB (no credits video) / 6.3GB (with credits video)

Masquerade
05-12-2021, 12:46
Grab oo2core from game files and rename from oo2core_8_win64.dll to oo2core_7_win64.dll.

oo2reck+srep+lolz/$mp4=srep

17.8GB ----> 13GB (no credits video) / 13.1GB (with credits video)

KaktoR
06-12-2021, 04:38
Chorus

12:00:47 - Selected ARC/DS method for Data1a-01.bin was: xt_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb27 3:lc8:mc1000000
12:00:47 - Selected ARC/DS method for Data1b-01.bin was: 1
12:00:47 - Selected ARC/DS method for Data1c-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
13:37:04 - Overall input size: 31.52 GB
13:37:04 - Overall output size: 24.28 GB (Ratio 77.03%)
13:37:04 - Overall conversion time: 01:36:12

For video files do not use bpk (crc error). Just use m1 or something similar.

dixen
08-12-2021, 14:13
Halo Infinite

*.module - oodle algorithm (kraken)

Decompressed fine with Xtool v0.39 + oo2core_8_win64.dll
28 gb > 60 gb

jihack
09-12-2021, 04:22
Hi! I'm new at game conversion. Is anyone know the best method to compress Sea of Thieves? And any tips for me as I am a beginner?

KaktoR
09-12-2021, 04:34
@jihack
Maybe precompress with zlib because of UE4 engine (most of cases).
Could be that the files are encrypted.

Send me a sample .pak file

Decryption key should be 0x37A0BC3DC2E01D9EB4923CA266A5701F56A4802347F07927 FC3FC25C93B31B50

jihack
09-12-2021, 04:41
@jihack
Maybe precompress with zlib because of UE4 engine (most of cases).
Could be that the files are encrypted.

Send me a sample .pak file

Decryption key should be 0x37A0BC3DC2E01D9EB4923CA266A5701F56A4802347F07927 FC3FC25C93B31B50

Yep I tried with some *.pak files with that eas key from rin but the ratio still 90%+. The tool I use is DiskPan GUI from Cesar82. I'll send the .pak file in pm.

jihack
09-12-2021, 05:02
@jihack
Maybe precompress with zlib because of UE4 engine (most of cases).
Could be that the files are encrypted.

Send me a sample .pak file

Decryption key should be 0x37A0BC3DC2E01D9EB4923CA266A5701F56A4802347F07927 FC3FC25C93B31B50

Well, here's the paks file https://drive.google.com/file/d/1A2yEcyRtpgBs18gAhgC2zGu8cMxMEecq/view?usp=sharing
Bcs I can't pm anyone

KaktoR
09-12-2021, 05:41
Maybe there is just "bad" data inside the files, hence the worse ratio.

KaktoR
09-12-2021, 07:47
Halo Infinite

12:29:36 - Selected ARC/DS method for Data1a-01.bin was: oo2reck:8+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb 273:lc8:mc1000000
12:29:36 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
----------------------------------------------------------------------------------------------------------------------------------------
16:44:16 - Overall input size: 48.00 GB
16:44:16 - Overall output size: 36.37 GB (Ratio 75.77%)
16:44:16 - Overall conversion time: 04:14:37

Data1a -> *.module, *.module_hd1
Data1b -> rest of the files

PS: You can rip languages from movie files
Academy_Tutorial_Intro.mp4 407mb -> 305mb with one language

KaktoR
09-12-2021, 14:27
Horizon Zero Dawn Complete Edition
v1.11, one language

19:32:12 - Selected ARC/DS method for Data1a-01.bin was: bpk
19:32:12 - Selected ARC/DS method for Data1b-01.bin was: oo2reck:3+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb 273:lc8:mc1000000
----------------------------------------------------------------------------------------------------------------------------------------
23:23:00 - Overall input size: 57.77 GB
23:23:00 - Overall output size: 41.74 GB (Ratio 72.25%)
23:23:00 - Data files creation time: 03:21:54

Exclude the following files from bpk

MQ1_Intro_at_the_Hovel.bk2
mq02_prepare.bk2
MQ06_07_DeathAndRebirth.bk2
MQ16_Epilogue1.bk2
MQ16_Victory.bk2
MQ030_YearsOfTraining.bk2
MQ050_ProvingCllimb.bk2

Dark0226
10-12-2021, 02:16
What algorithms should be used to process SpellForce 3: Reforced?

shazzla
10-12-2021, 03:16
Good ideas to start :

1. Look for compression/encryption algorithm names in folder structure.
2. In Total Commander ,view the main gamefiles and its DLLs (F3) and search for comp/encr-algonames,too.
3. If none of them found ,try zlib ,maybe it works.
4. If zlib doesnt work,give it up!:D

KaktoR
10-12-2021, 03:30
For dds, you can use just srep+lolz
ogv are video files, try to use a low freearc setting (m1, m2) or just use m0 for storing (sometimes you can compress video files, but just a few mb)
lvl, crf I don't know

As for "bin_addons_win32/SF3_Campaign/package.pak_data" you can try zlib

fabrieunko
10-12-2021, 04:49
Hello what would be the best method for icarus? please. thanks in advance

Masquerade
10-12-2021, 08:51
Hello what would be the best method for icarus? please. thanks in advance

srep+lolz/lzma - devs didn't use compression on their paks.

hwang4
15-12-2021, 00:29
hi! need a valid unreal key ! :)

Masquerade
15-12-2021, 04:35
hi! need a valid unreal key ! :)

What do you mean? This game isn't encrypted.

hwang4
15-12-2021, 04:50
sorry !thought it was!:)

Masquerade
17-12-2021, 04:13
ARK Survival Evolved

400GB ----> 38.7GB

srep+lolz
mpz on soundtrack

Compression takes almost 40 hours, not for the faint hearted.

KaktoR
17-12-2021, 04:33
400GB... I guess this game reaches 1TB soon :D

KaktoR
17-12-2021, 05:06
Final Fantasy VII Remake Intergrade

oodle kraken 7

pakchunk0_s23-WindowsNoEditor.pak

Compressed 1 file, 8,184,822,018 => 8,551,069,519 bytes. Ratio 104.47%
Compression time: cpu 4.52 sec/real 1049.03 sec = 0%. Speed 7.80 mB/s
All OK

Video files are webm videos

dixen
17-12-2021, 09:32
Final Fantasy VII Remake Intergrade

pakchunk0_s5-WindowsNoEditor.pak

Compressed 1 file, 2,076,132,806 => 8,448,777,571 bytes. Ratio 406.95%
Compression time: cpu 1.80 sec/real 492.10 sec = 0%. Speed 4.22 mB/s
All OK

Tested 1 file, 8,448,777,571 => 2,076,132,806 bytes. Ratio 406.95%
Testing time: cpu 2.11 sec/real 344.89 sec = 1%. Speed 6.02 mB/s
All OK

XTool v0.39 + oo2core.dll from game folder

KaktoR
17-12-2021, 11:25
Anyone knows what's the point for all the different language movie files in FF7 Remake? I guess only lips movement right?

Masquerade
17-12-2021, 14:21
Anyone knows what's the point for all the different language movie files in FF7 Remake? I guess only lips movement right?

Yes, US files are for EN and all else is for FR/DE/JP - there are some dedicated videos for FR/DE videos (e.g. credits video).

P.S. do not rip any credits videos since the game actually has an ending scene before the credits show. If you rip the credits video, you lose the ending of the game.

KaktoR
18-12-2021, 04:37
Ok then I guess you can rip all the lipsync files out of s21 pak and voices from s23 pak

Masquerade
18-12-2021, 05:44
Ok then I guess you can rip all the lipsync files out of s21 pak and voices from s23 pak

I've already ripped out the FR, DE, JP voices but didn't touch the lipsync. I cannot imagine it would save much space?

KaktoR
18-12-2021, 06:13
Well I didn't extract the archive yet to see how much size they have.

KaktoR
18-12-2021, 15:37
Assassin's Creed Odyssey
v1.5.6, One language

14:00:49 - Selected ARC/DS method for Data1a-01.bin was: oo2recm:4+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb 273:lc8:mc1000000
14:00:49 - Selected ARC/DS method for Data1b-01.bin was: bpk+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc 8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
00:35:56 - Overall input size: 99.23 GB
00:35:56 - Overall output size: 40.75 GB (Ratio 41.06%)
00:35:56 - Data files creation time: 10:04:52

First archive *forge files
Second archive rest of the files

panker1992
19-12-2021, 04:14
Ok then I guess you can rip all the lipsync files out of s21 pak and voices from s23 pak

how do you unpack those pak files with oodle support ??

do you know the blocks for languages to rip via hex editor ?

Masquerade
19-12-2021, 04:17
how do you unpack those pak files with oodle support ??

do you know the blocks for languages to rip via hex editor ?

https://anonfiles.com/V9C101q5wb/unreal_tournament_4_0.4.25b_bms

quickbms -l "unreal_tournament_4_0.4.25b.bms" pakchunk0_s23-WindowsNoEditor.pak >> pakchunk0_s23-WindowsNoEditor.txt

This lists the contents and the locations of all the files in the pak.

German Block: 2583541977 - 4031342431

French Block: 4031342793 - 5445490559

Japanese Block: 5445490905 - 6823955647

KaktoR
19-12-2021, 06:03
Assassin's Creed Valhalla
Changed from oodle mermaid 7 to 8 in v1.4.1

oo2core_7_win64.dll
Compressed 1 file, 1,103,790,080 => 1,747,822,450 bytes. Ratio 158.35%
Compression time: cpu 1.30 sec/real 492.31 sec = 0%. Speed 2.24 mB/s
All OK

oo2core_8_win64.dll
Compressed 1 file, 1,103,790,080 => 1,776,145,142 bytes. Ratio 160.91%
Compression time: cpu 1.20 sec/real 507.94 sec = 0%. Speed 2.17 mB/s
All OK

KaktoR
19-12-2021, 08:59
Anyone knows how to use bsc with freearc? Packing works, but unpacking doesn't work here

panker1992
20-12-2021, 03:12
https://anonfiles.com/V9C101q5wb/unreal_tournament_4_0.4.25b_bms

quickbms -l "unreal_tournament_4_0.4.25b.bms" pakchunk0_s23-WindowsNoEditor.pak >> pakchunk0_s23-WindowsNoEditor.txt

This lists the contents and the locations of all the files in the pak.

German Block: 2583541977 - 4031342431

French Block: 4031342793 - 5445490559

Japanese Block: 5445490905 - 6823955647


This doesn't work at all, it asks for a key!

thanks for blocks :D

KaktoR
20-12-2021, 04:02
This doesn't work at all, it asks for a key!

thanks for blocks :D

The key is
0x23989837645C9D28BA58072B2076E895B853A7C9E1C5591B 814C4FD2A2D7B782

panker1992
20-12-2021, 04:19
thanks alot :D

lipsync isnt worth the hassle... 30 mb and if you compress it you will gain 1 or 2 mb of space

removing voice though saves 1.4 GB per language

Masquerade
20-12-2021, 04:35
panker1992
I thought you would be capable of finding the key yourself :p

There's a list of keys and an AES finder application uploaded by Spiritovod on the CounterStrike.RussianInformationNetwork ;) which I cannot link here for obvious reasons. You can use those resources in future.

KaktoR
20-12-2021, 04:38
Or use this, if you don't have an account over there (I update this list every week for DSG)
https://drive.google.com/file/d/1aF30x5r3NXqK45piS56b59yxNyltVH4V/view?usp=sharing

ffmla
20-12-2021, 09:25
Anyone knows how to use bsc with freearc? Packing works, but unpacking doesn't work here

in arc.ini
[External compressor:bsc]
header = 0
packcmd = bsc.exe e $$arcdatafile$$.tmp $$arcpackedfile$$.tmp {options}

For Pack
arc.exe a -ep1 -dses --dirs -s; -lc- -di -i2 -r -m=bsc:b128:m0 data_bsc.arc _Pack\*

For Unpack

arc.exe x -dpunpacked\ data_bsc.arc

NOTE:
cls version (Shegorat) available in Krinkels for 0.3.1

KaktoR
20-12-2021, 09:37
Thanks, I already know about this, but was corious if it is possible with bsc.exe too, but it doesn't looks so.

panker1992
20-12-2021, 14:42
panker1992
I thought you would be capable of finding the key yourself :p

There's a list of keys and an AES finder application uploaded by Spiritovod on the CounterStrike.RussianInformationNetwork ;) which I cannot link here for obvious reasons. You can use those resources in future.

I have a very tight schedule these days my friend,
anything that cant be automated or i dont solve in under 5 minutes goes to bin.

BTW Final Fantasy is the worst remake for these past years according to some !!!

let me at least share my findings before i delete this game

ffmpeg -i <emov.file> -c:v libvpx-vp9 -b:v 2000k -c:a copy -c:s copy -y <output.webm_then_rename>

use this on all videos

you will end up with 2.2GB of videos

dixen
20-12-2021, 20:26
Five Nights at Freddy’s: Security Breach (Build 7902251)

*.pak - ue4d+srep+lolz
75 gb > 54 gb > 28.5 gb

*.mp4 - srep+zstd
no changed size (3.4 gb)

Rest files - srep+lolz

Init encryption key: 85F7D4007015493ED0359C9007266038F8F7B1F96988F19A61 0103874CC95286
99.8%
Total encrypted UE records 46345/46383
Tested 1 file, 30,685,710,536 => 81,036,615,981 bytes. Ratio 37.87%
Testing time: cpu 239.63 sec/real 434.69 sec = 55%. Speed 186.42 mB/s
All OK

Thanks to Masquerade for update & Shegorat for ue4d plugin.

L33THAK0R
22-12-2021, 18:18
*.mp4 - srep+zstd
no changed size (3.4 gb)

Can I ask why you used zstd for the "*.mp4" files?

Masquerade
23-12-2021, 00:21
Can I ask why you used zstd for the "*.mp4" files?
ZSTD is very fast and you can maybe squeeze compression ratio to 98%.

dixen
23-12-2021, 01:22
Can I ask why you used zstd for the "*.mp4" files?

Fast unpack - ~300-800 Mbit on uncompressed data (movies, audio and encrypted data)

Masquerade
23-12-2021, 13:42
Ripped Credits Video

XTool:zlib+srep+lolz

Videos: srep+zstd

12.1GB ---> 7.6GB

KaktoR
24-12-2021, 16:00
Assassin's Creed Valhalla
v1.4.1, all DLC, One language

*forge files -> oo2recm+srep+lzma-mt (use oo2core_8_win64.dll rename to 7)
rest of the files -> srep+lzma-mt

97.1GB -> 40.9GB
Time: 10 hours

PS: DSG just crashed at ISO creation and no LOG was saved.

KaktoR
29-12-2021, 17:56
Final Fantasy VII Remake Intergrade
Nothing ripped

13:26:23 - Selected ARC/DS method for Data1a-01.bin was: oo2reck:7+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb 273:lc8:mc1000000
13:26:23 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
13:26:23 - Selected ARC/DS method for Data1c-01.bin was: 1
-------------------------------------------------------------------------------------------------------------------------------------------
00:55:09 - Overall input size: 94.31 GB
00:55:09 - Overall output size: 82.01 GB (Ratio 86.95%)
00:55:09 - Overall conversion time: 11:27:39

Data1a -> pak files
Data1b -> all except pak and emov
Data1c -> emov files

You can save additional ~2GB with removing languages from pak files.
https://fileforums.com/showpost.php?p=495210&postcount=2701

:( Sad8669
31-12-2021, 06:56
Method : srep:m3f:l512+lolz:dtb1:d128:mtt0:mt4:mc1023
Method (on USM files) : srep:m3f:l512+zstd
Compression Time : 7.5 Hours
Decompression Time : Soon
Initial Size : 34.1 GB
Final Size : 16.2 GB
Extra Tip : USM's are worth 5.09 GB and can be encoded to almost half of its original size but you know i am just too lazy for that.


Side : It's been a while since i have posted, i am just lazy lol. Anyway good to see you guys digging on newest games.

Masquerade
31-12-2021, 10:05
msc+xtool_zlib+srep+lolz

4.7GB ---> 1.6GB

:( Sad8669
01-01-2022, 23:24
Method : srep:m3f:l512+lolz:dtb1:d128:mtt0:mt4:mc1023
Compression Time : 9.2 Hours
Decompression Time : 7.7 Minutes
Initial Size : 25.1 GB
Final Size : 11.3 GB
Extra Tip : USM's are worth 2.22 GB and can be encoded to almost half of its original size.

KaktoR
05-01-2022, 05:29
Anno 1404 History Edition
Nothing ripped, all languages

Selected ARC/DS method for Data1a-01.bin was: srep_new+mpz_mtx
Selected ARC/DS method for Data1b-01.bin was: srep_new+msc_tak+4x4:b64mb:lzma:ultra:64m:bt4:fb27 3:lc8:mc1000000
Selected ARC/SPLIT method for Data1c-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000/$mp3pack=srep_new+mpz_mtx/$jpgpack=packjpg
-----------------------------------------------------------------------------------------
Installation folder size: 7.26 GB
Compressed game size: 4.47 GB (Ratio 61.52%)

Data1a = all language files in addon/maindata folder
Data1b = maindata\data4.rda
Data1c = rest of the files including Soundtrack/Poster

Note: maindata\data4.rda has bink videos inside. You can compress with bpk, but unpacking will lead to crc errors.

Anno 1503 History Edition
Nothing ripped, all languages

Selected ARC/SPLIT method for Data2a-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000/$bpkpack=bpk/$mscpack=srep_new+msc_tak/$mp3pack=srep_new+mpz_mtx/$jpgpack=packjpg
-----------------------------------------------------------------------------------------
Installation folder size: 6.75 GB
Compressed game size: 3.32 GB (Ratio 49.25%)

Anno 1602 History Edition
Nothing ripped, all languages

Selected ARC/SPLIT method for Data3a-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000/$bpkpack=1/$mscpack=srep_new+msc_tak/$mp3pack=srep_new+mpz_mtx/$jpgpack=packjpg
-----------------------------------------------------------------------------------------
Installation folder size: 1.52 GB
Compressed game size: 0.94 GB (Ratio 61.83%)

Note: Using binkpack for bik videos will lead to crc errors on all files if you unpack.

Anno 1701 History Edition
Nothing ripped, all languages

Selected ARC/SPLIT method for Data4a-01.bin was: xt_zlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb27 3:lc8:mc1000000/$anno1701hempz=bpk+srep_new+mpz_mtx/$anno1701hebpk=bpk+srep_new+4x4:b64mb:lzma:ultra:6 4m:bt4:fb273:lc8:mc1000000/$mp3pack=srep_new+mpz_mtx/$jpgpack=packjpg/$libpack=srep_new+razor_mtx
-----------------------------------------------------------------------------------------
Installation folder size: 12.77 GB
Compressed game size: 5.86 GB (Ratio 45.87%)

Mask $anno1701hempz = data2_<language>.rda (all of this files also containing bink videos. binkpack works fine here)
Mask $anno1701hebpk = data4.rda, data5.rda (this files also containing non-bink files, so use bpk+<some compressor>)

Overall Note: For the rda files in the "ahe" folders of all 4 games you can use zlib

NOTE:
Anno 1404 History Edition
maindata\data4.rda (contains bink videos/wav/raw/mp3, bpk crc error, msc works fine)

Anno 1503 History Edition
Videos\1080p_30fps_5Sec.bk2 (bpk crc error)
Videos\1080p_30fps_5Sec.bk2 (bpk crc error)
BonusContent\Anno_1503_soundtrack\60 - Windgames.mp3 (mpz unpack error, use packmp3/msc)
ToolGfx\*.bmp (msc crash)

Anno 1701 History Edition
data2_<lang>.rda (contains bink videos/mp3, bpk+mpz throws errors, use bpk+msc)

L33THAK0R
05-01-2022, 09:28
Has anyone had any luck with "Devil May Cry 4: Special Edition"? Tried what feels like every version of XTool 2019/2020 under the sun in conjunction with srep & lolz but the ".arc" archives always seem to fail to extract properly on decompression.

:( Sad8669
05-01-2022, 09:47
May i know the full description of the error?

If not try sending me a very small sample of that game*arc file with mentioned compression applied, i will look into it.

L33THAK0R
07-01-2022, 01:59
May i know the full description of the error?

If not try sending me a very small sample of that game*arc file with mentioned compression applied, i will look into it.

Cheers mate, but I've managed to resolve my issue. I was fiddling with lolz a tad and it seems perhaps the culprit was the dictionary size used, after lowering the size of said dictionary decompression was successful. I wasn't aware this setting could jeopardise the integrity of an archive, which might explain a couple of other compression I've attempted ("Deus Ex: Mankind Divided" & The first "Nioh" title) but failed to achieve successful decompression of their respective packed archives.

Masquerade
07-01-2022, 09:15
I've not encountered any such error with my repacks ^^

Snake288
07-01-2022, 09:23
Watch Dogs Legion hi how do I do the compression process

KaktoR
07-01-2022, 10:12
As for WDL, you can straight pack videos_ultra.dat with bpk. For videos.dat this does not work. It get stuck on 6.4%, I guess you have to unpack the file first.

Compressed 1 file, 3,704,538,840 => 3,145,628,556 bytes. Ratio 84.91%
Compression time: cpu 2.45 sec/real 401.17 sec = 1%. Speed 9.23 mB/s
All OK

Extracted 1 file, 3,145,628,556 => 3,704,538,840 bytes. Ratio 84.91%
Extraction time: cpu 4.22 sec/real 367.05 sec = 1%. Speed 10.09 mB/s
All OK

its_gnar_again
07-01-2022, 10:20
I am having an issue with UELR, it starts to do it's thing, then stops with this error:

ERROR: general (de)compression error in uelr

Here's my setup:

[External compressor:uelr]
header = 0
packcmd = UELR\uelr_x64 uv $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

I have also tried using "ulv" rather than "uv" but it brings up the error still. The game I am using it on is Lost Planet 3 BTW.

KaktoR
07-01-2022, 11:12
I am having an issue with UELR, it starts to do it's thing, then stops with this error:

ERROR: general (de)compression error in uelr

Here's my setup:


I have also tried using "ulv" rather than "uv" but it brings up the error still. The game I am using it on is Lost Planet 3 BTW.

Make sure to only process tfc/upk files with uelr

Masquerade
07-01-2022, 11:44
UELR will crash out if no LZO streams are detected.

fabrieunko
07-01-2022, 12:55
Hello, what method should we use to compact MP3 files?

Masquerade
07-01-2022, 13:37
Hello, what method should we use to compact MP3 files?
https://bayfiles.com/F6k9Jc52x4/MPZ_rar

its_gnar_again
07-01-2022, 15:42
Make sure to only process tfc/upk files with uelr


I separated ONLY tfc/upk files, and tried again using UELR, and I get the same error. (uelr+srep+lolz/xz should be correct, right? I see other methods using this just fine.)

UELR will crash out if no LZO streams are detected.

GFS does show LZO streams, and while attempting to compress with UELR it does seem like its detecting them/working before crashing :(

:( Sad8669
08-01-2022, 00:58
KakTor

There is a tool named WDLExtract if i recall correctly.

last time i checked the filelist wasn't complete and it was abandoned as far as i know, but videos.dat is complete. BPK works on most of them (Not sure about decompression) except for few binks and 4K one's can be ripped.

KaktoR
08-01-2022, 03:56
KakTor

There is a tool named WDLExtract if i recall correctly.

last time i checked the filelist wasn't complete and it was abandoned as far as i know, but videos.dat is complete. BPK works on most of them (Not sure about decompression) except for few binks and 4K one's can be ripped.

I will test it and report back later, thanks.

https://forum.xentax.com/viewtopic.php?f=10&t=22903#p168208 + https://forum.xentax.com/viewtopic.php?f=10&t=22903&start=30#p168732

For repack the archive https://github.com/rootCBR/PackLegion

KaktoR
08-01-2022, 05:14
Tests are done.

There are some *-pcultra.bik files inside videos.dat file (3,15 GB). These files have higher resolution (4K) but have almost the same size as the normal videos (FullHD), makes me wonder this sorcery. I don't know if you can rip them safely, but I guess so.

As already said, you can directly compress videos_ultra.dat file with binkpack. No need to unpack this file.

Here are the files for which binkpack will NOT work (motion_data warn, crc check failed on unpacking).
grapics\bink\wd3_alpha_fs\* (NOTE: Just some small videos, not worth the hasle to trial&error file by file)
grapics\bink\wd3_alpha_os\rds_210_s001_vid_intro_o s.bik
grapics\bink\wd3_bagleys\gst_003_s040_bb_new_team. bik
grapics\bink\wd3_bagleys\opl_108_s010_bb_emb_exp.b ik
grapics\bink\wd3_bagleys\opl_321_s010_bb_bagleybri efing.bik

Batch for WDLExtract
WdlExtract.exe <Input fat file> <Output Folder>

WdlExtract.exe "E:\test\videos.fat" "E:\test\videos_unpacked"

Batch for PackLegion
PackLegion.exe <Input Folder> <Output fat file>

PackLegion.exe "E:\test\videos_unpacked" "E:\test\videos_new.fat"

CRC checks
videos.dat 40ff9a3b
videos.fat 9d689549
videos_new.dat 8477a943
videos_new.fat a4915500

I don't know if the game will work correctly with "bad crc" on a game archive.

Edit: Tested and riped all *-pcultra.bik files out of videos.dat. The game works fine and videos will be play normally (1080p/1440p resolution).

Masquerade
09-01-2022, 00:48
SWORD x HIME - v1.82

srep+lolz

11.7GB ---> 3.6GB

Masquerade
09-01-2022, 05:59
From The Depths: v3.4.2

srep+lolz/$wav=msc+srep+LZMA2

3.2GB ---> 1.4GB

Masquerade
09-01-2022, 06:02
Blood Bond – Into the Shroud [Enhanced Edition]

xtool+srep+lolz/$mp4=srep+zstd

6.6GB ---> 4.5GB

KaktoR
09-01-2022, 07:26
Time for an oldie

Lost Planet Extreme Condition

16:07:43 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4: fb273:lc8:mc1000000
16:07:43 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
-------------------------------------------------------------------------------------------------------------------------------------------
16:24:14 - Overall input size: 7.89 GB
16:24:14 - Overall output size: 2.65 GB (Ratio 33.62%)
16:24:14 - Overall conversion time: 00:16:29

Lost Planet 2

16:34:19 - Selected ARC/DS method for Data1a-01.bin was: xtool:mzlib+srep_new+4x4:b64mb:lzma:ultra:64m:bt4: fb273:lc8:mc1000000
16:34:19 - Selected ARC/DS method for Data1b-01.bin was: srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc 1000000
-------------------------------------------------------------------------------------------------------------------------------------------
16:59:31 - Overall input size: 11.79 GB
16:59:31 - Overall output size: 4.10 GB (Ratio 34.78%)
16:59:31 - Overall conversion time: 00:25:07

Lost Planet 3

12:18:22 - Selected ARC/DS method for Data1a-01.bin was: uelr+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:l c8:mc1000000
12:18:22 - Selected ARC/DS method for Data1b-01.bin was: bpk+srep_new+4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc 8:mc1000000
-------------------------------------------------------------------------------------------------------------------------------------------
13:56:53 - Overall input size: 22.27 GB
13:56:53 - Overall output size: 11.04 GB (Ratio 49.57%)
13:56:53 - Overall conversion time: 01:38:24

Masquerade
10-01-2022, 00:40
Nordic Warriors

xtool+srep+lolz/$mp4=srep+zstd

19.8GB ---> 7.3GB

Masquerade
14-01-2022, 00:15
Be careful if you use my data blocks to patch out language files for Final Fantasy Remake Integrade. I think I might have made an error with the Japanese block which is causing fatal crashes deep in the game.

abidnnt
14-01-2022, 11:42
Can anyone share what method will be best for God Of War - FLT? TIA.

Masquerade
14-01-2022, 12:15
God Of War: a quick look.

Main: srep+lolz

BPK on the videos (yes, it works!)

srep+razor on audio

Good luck friends.

KaktoR
14-01-2022, 14:03
Are there really 100k files? Or steamdb is trolling

Masquerade
14-01-2022, 14:11
Are there really 100k files? Or steamdb is trolling

Yep, over 150k wems.

fabrieunko
14-01-2022, 20:39
Hello, I have a question about the use of wemtool once launched with the -e command and that we obtained the .DIFFZ files how to return to the WEM files? I understood that it was necessary to do -d . how do you use wemtool once you have compacted everything in bin? the installer does not launch wemtool -d?.. how to do? is there a place that talks about how to use it?
thank you very much

I tested wemtool on god of war. there is a sound directory with a french directory in which there are 12349 wem files. I launched WemToll -e it finished and erase all the files wem.. to test I launched Wemtool -d it finished and erase all the DIFF. but now i only have 2725 wem files. Why?

Masquerade
14-01-2022, 23:15
I'll check that out, it could be bugged.

Furjack
15-01-2022, 00:07
what is compression method for god of war pc

KaktoR
15-01-2022, 00:22
what is compression method for god of war pc

Read a page back

Furjack
15-01-2022, 06:38
Read a page back

Thanx:):D