Best Compression Methods for 'Specific' Games. Q&A

[KaktoR]

Some test for AC Liberation Remastered
(use afr without -a2)

DataPC_ACV_Chichen_Itza.forge
301.596.672 Bytes

afr+srep:m3f+

lolz:dtb1:mtt1:mt6:d128m:mc1023
Compressed 1 file, 301,596,672 => 24,798,849 bytes. Ratio 8.22%
Compression time: cpu 0.89 sec/real 245.85 sec = 0%. Speed 1.23 mB/s
All OK

4x4:b64mb:lzma:ultra:64m:bt4:fb273:lc8:mc1000000
Compressed 1 file, 301,596,672 => 27,686,068 bytes. Ratio 9.18%
Compression time: cpu 0.73 sec/real 55.38 sec = 1%. Speed 5.45 mB/s
All OK

lzma:a1:mfbt4:d256m:fb273:mc1000000000:lc8
Compressed 1 file, 301,596,672 => 27,596,500 bytes. Ratio 9.15%
Compression time: cpu 0.36 sec/real 50.47 sec = 1%. Speed 5.98 mB/s
All OK

zstd -22 --ultra -T2
Compressed 1 file, 301,596,672 => 31,890,007 bytes. Ratio 10.57%
Compression time: cpu 0.33 sec/real 91.99 sec = 0%. Speed 3.28 mB/s
All OK

Edit:

ACIII:R+L:R

Compressed block count (subblocks) = 353022 (2186139)
LZO1X-W3(15) subblocks count       = 1480503
LZO1X-999 subblocks count          = 0
LZO2A-999 subblocks count          = 0
LZO1C-999 subblocks count          = 0
OODLE subblocks count              = 0
ZSTD  subblocks count              = 0
Uncompressed subblocks count       = 705636

Compressed subblocks size:   19522484316 -> 44694970389
Uncompressed subblocks size: 17468349094
Other data size:             315047570

Total data+table:            37314625536 -> 62498687973

WARNING! In file found corrupted LZO-streams. Check your data!

Time: 2967.089 secs

Edit:

17:42:22 - Starting Data1.bin.001 creation
23:41:59 - Finished Data1.bin.001 creation
23:42:00 - Starting Data2.bin.001 creation
 0:50:03 - Finished Data2.bin.001 creation
 Input size:  39.510g
 Output size: 13.537g

[darkwolves]

method
decrypted ttarch2 files
after decryption they contained zlib streams
precompressed with Xtool By Razor12911
Xzlib+srep+lolz:dt1:dtp1:dto1:dtm1:dtw1:dtd1:mtt1: mt4:d200m:tt4 h14:fba4096:mc1023:bc8:blr8:bm
initial size 13.7 gb
repack size 6.28 gb

[danswano]

Originally Posted by darkwolves

method
decrypted ttarch2 files
after decryption they contained zlib streams
precompressed with Xtool By Razor12911
Xzlib+srep+lolz:dt1:dtp1:dto1:dtm1:dtw1:dtd1:mtt1: mt4:d200m:tt4 h14:fba4096:mc1023:bc8:blr8:bm
initial size 13.7 gb
repack size 6.28 gb

What is the size of precompressed file?

[darkwolves]

Originally Posted by danswano

What is the size of precompressed file?

if i am being honest i do not know i had it repacking when i was at work

[danswano]

Originally Posted by darkwolves

if i am being honest i do not know i had it repacking when i was at work

Can you upload the tool you used to decrypt the files please?

[darkwolves]

Originally Posted by danswano

Can you upload the tool you used to decrypt the files please?

certainly
credit goes to Luigi Auriemma & Edison007 for the tool


======note
this tool works on
"The Wolf Among Us"
"The Walking Dead: Season 2" "Tales from the Borderlands (all episodes)"
"Game of Thrones (all episodes)"
"Minecraft: Story Mode"
"The Walking Dead: Michonne"
"Batman: The Telltale Series"
"The Walking Dead: A New Frontier"
"Marvel's Guardians of the Galaxy"
"Minecraft: Story Mode - Season Two"
"Batman: The Enemy Within"
"The Walking Dead: The Final Season"

[Kitsune1982]

Hi all, just wanted to post a brief comment about the old XT.exe b in the _Tools (2019-01-22).7z vs the one in _Tools (2019-03-28).7z, XT.exe

I experimented a little bit on some test files and this is what i discovered.

The game I compressed, Agarest Generations of War took 30 minutes to decompress on a dual core AMD cpu with 8 gb ram and 4 gb vram/1 gb dedicated video. while this might sound normal, i found out something interesting.

the original compression method i used with UltraArc 2900 R6 Full/_Diskpan.bat folder from the Tools (2019-01-22).7z for the second and third set of .bra files was xZ1+srep:m5f+msc+lolz:d256m:mc1023. for the 1st folder which was all videos i just used srep:m5f+LZMA. the third set of files i used the same method as the second set, and the total decompression time was 30 minutes using the old XT/xtool from that package i specified above.

now heres the interesting part. i downloaded and extracted the compression tools from kaktors _Tools (2019-03-28).7z file on his CIUV3 thread, and copy/pasted the largest .bra file which just has .ogg music files stored in it to a seperate test folder, and after modifiying my ciuv3 script a bit to just process the test.bin.001 file i made with diskspan, i recreated the same installer with just the one fille then ran the setup again. this time the installer unpacked the data from that new text .bin file in 1 minute!! the same file that it got stuck on for like 15 minutes from my first compression test unpacked in only one minute. :O the only change i made to the fa.ini script in the ultraarc 2900 r5 FULL/resources directory was changing the decomp setting under xzlib from 100p to t40.

what i dont understand is that i did the same experiment with the compression using the same settings with the old xtool program from the older _tools archive from 1-2019 and it was 3x longer, it took 30 min to install the game where just the newe one unpacked almost a gb of data in under a minute, and on a dual core desktop to boot. moreover when i was on my quad core laptop with 8 gb ram and a 4 gb vram integrated video card, the decompression took about 8 minutes less but it still was almost 20 nin to fully install a 4.7 gb game.

as a final comment, i did use lolz to compress the .bra files along with cxlib but even the lolz compressor processed quickly, it wasnt until the old xtool started prcoessing the last of the files that it hung for like 15 min on a 90 mb file.

needless to same, im pleased with the speed increase. any thoughts or ideas about why an older one was slower than the newer one? only thing that comes to mind is the xtool in the new package was a 64 bit version and the older one was 32 bit. let me know what you all think.

-Kitsune1982

"They couldnt hit an elephant at this dis---" last words of general custer during the civil war.

Addendum Note - 4/11/19

Ok, i tried another experiment. i repacked the game's .bra files that had been giving me trouble
with the new tools using the default settings UltraArc r2900 Full's diskspan folder has, and let it compress. it took a bout an hour or so, i dont reallyknow how long because i started the compression then went to sleep. when i woke up, i edited the setup.ini file in the ciuv3 folder and attempted to reinstall
the game again. this time the whole game took 10 minutes to decompress!!! So for whatever reason, it seems the updated XT.exe file in the recent tools package
Kaktor put together is much faster and more stable than the old one. this is the method i used.

XZ1+srep:m5f+msc+lolz:d512m:mc1023, and left packcmd at t75p and chunk size at 32mb. I have confirmed the newer one works faster and doesnt freeze up slower computers
as much for some reason. In any case, I know which one im using now. leave any feedback below this post.

thanks, have a good one.

-Kitsune1982

"they couldnt hit an elephant at this dis---" famous last words of general custer from custer's last stand during the civil war.
guess those musket rifles can hit even an elephant at that distance.

[doofoo24]

generic setting

[External compressor:srep]
header = 0
packcmd = srep.exe -m3f -d1g -a2 $$arcdatafile$$.tmp - <stdout>

[External compressor:lolz]
header = 0
packcmd = lolz.exe -mt4 -mtt1 -d64 -fba4096 -mc1023 -pc4 -gm20 -dtb1 -x2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

[External compressor:xprecomp]
header = 0
packcmd = xtool.exe erecomp:c256mb,t4:zlib - - <stdin> <stdout>

*as a side note, for lolz change mt to match cpu core and for xtool no need to change cm to smaller number it will make xtool slower during installation beside "BIGGER ALWAYS BETTER"

[Kitsune1982]

Agarest Generations of War Zero Updated 4-16-19 (4.55 GB)

Repack Name: Agarest Generations of War Zero
Compression Methods used: long version:
xZLib, srep64. lolz 64 bit
short version: xZ1+srep:m5f+lolz:d256m:mc1023
Media Ripped/Removed/Re-encoded: Nothing!
Original Installed Size: 5.05 gb

Compressed Output Size - 5.0 gb -> 4.5 gb

Output file names:

Wolfpaw-1.bin.001
Wolfpaw-2.bin.001
Wolfpaw-3.bin.001
Wolfpaw-4.bin.001

Compression Time and specifications of hardware used:
Dual Core @ 2.90 ghz - 8 GB RAM 4 GB Vram
Decompression TIme: 10 Min on 2 cores, 5-10 minutes on 4+

[KaktoR]

Originally Posted by Kitsune1982

Hi all, just wanted to post a brief comment about the old XT.exe b in the _Tools (2019-01-22).7z vs the one in _Tools (2019-03-28).7z, XT.exe

I experimented a little bit on some test files and this is what i discovered.

The game I compressed, Agarest Generations of War took 30 minutes to decompress on a dual core AMD cpu with 8 gb ram and 4 gb vram/1 gb dedicated video. while this might sound normal, i found out something interesting.

the original compression method i used with UltraArc 2900 R6 Full/_Diskpan.bat folder from the Tools (2019-01-22).7z for the second and third set of .bra files was xZ1+srep:m5f+msc+lolz:d256m:mc1023. for the 1st folder which was all videos i just used srep:m5f+LZMA. the third set of files i used the same method as the second set, and the total decompression time was 30 minutes using the old XT/xtool from that package i specified above.

now heres the interesting part. i downloaded and extracted the compression tools from kaktors _Tools (2019-03-28).7z file on his CIUV3 thread, and copy/pasted the largest .bra file which just has .ogg music files stored in it to a seperate test folder, and after modifiying my ciuv3 script a bit to just process the test.bin.001 file i made with diskspan, i recreated the same installer with just the one fille then ran the setup again. this time the installer unpacked the data from that new text .bin file in 1 minute!! the same file that it got stuck on for like 15 minutes from my first compression test unpacked in only one minute. :O the only change i made to the fa.ini script in the ultraarc 2900 r5 FULL/resources directory was changing the decomp setting under xzlib from 100p to t40.

what i dont understand is that i did the same experiment with the compression using the same settings with the old xtool program from the older _tools archive from 1-2019 and it was 3x longer, it took 30 min to install the game where just the newe one unpacked almost a gb of data in under a minute, and on a dual core desktop to boot. moreover when i was on my quad core laptop with 8 gb ram and a 4 gb vram integrated video card, the decompression took about 8 minutes less but it still was almost 20 nin to fully install a 4.7 gb game.

as a final comment, i did use lolz to compress the .bra files along with cxlib but even the lolz compressor processed quickly, it wasnt until the old xtool started prcoessing the last of the files that it hung for like 15 min on a 90 mb file.

needless to same, im pleased with the speed increase. any thoughts or ideas about why an older one was slower than the newer one? only thing that comes to mind is the xtool in the new package was a 64 bit version and the older one was 32 bit. let me know what you all think.

-Kitsune1982

"They couldnt hit an elephant at this dis---" last words of general custer during the civil war.

Addendum Note - 4/11/19

Ok, i tried another experiment. i repacked the game's .bra files that had been giving me trouble
with the new tools using the default settings UltraArc r2900 Full's diskspan folder has, and let it compress. it took a bout an hour or so, i dont reallyknow how long because i started the compression then went to sleep. when i woke up, i edited the setup.ini file in the ciuv3 folder and attempted to reinstall
the game again. this time the whole game took 10 minutes to decompress!!! So for whatever reason, it seems the updated XT.exe file in the recent tools package
Kaktor put together is much faster and more stable than the old one. this is the method i used.

XZ1+srep:m5f+msc+lolz:d512m:mc1023, and left packcmd at t75p and chunk size at 32mb. I have confirmed the newer one works faster and doesnt freeze up slower computers
as much for some reason. In any case, I know which one im using now. leave any feedback below this post.

thanks, have a good one.

-Kitsune1982

"they couldnt hit an elephant at this dis---" famous last words of general custer from custer's last stand during the civil war.
guess those musket rifles can hit even an elephant at that distance.

This is not surprising.

Packing a file, which contains only audio (ogg) files with XTool doesn't make sense. XTool has nothing to work during unpacking audio files (recompression) because XTool had nothing to work on precompression, and therefore it is fast.

You should test your theories with zips, png, pdf or other files which contains streams (as large as possible) to get a accurate time.

[Kitsune1982]

Hi Kaktor. I forgot to mention I also used this same method on the other files which contain data and dds texture as well as other assets such as text, models and so on. the ogg file test was simply that. just a test to see how fast it would extract. after applying the recompression to the other files, the game installed much faster than before. before the whole game took 30 min to install, and after i recompressed the .bra files which contain data textures and other assets with lolz and msc which can compress dds wav and mp3 files, then recompiled the installer, the game installed completely in 10 minutes. just thought i would mention that.

-Kitsune