How can I compress much smaller than LZMA, that I want repack games?

[Unjk223]

Hello, everybody. I have problem with repack games.

Explanation:
I want to compress smaller than 7z, zip, rar, because I want to make same repack size smaller than others, let's say that I want compress Hedon.ipk3 original file to my repack file (.bin).

Here are about insignificant details: Percent is ratio.
Original before compress: 595 136 KB (100%)
LZMA: 560 360 KB (94%)
Zstandard: 555 498 KB (93%)
Brotli: 555 365 KB (93%)
LZ4: 562 922 KB (95%)
LZ5: 564 075 KB (95%)
Lizard, FastLZ4: 563 373 KB (95%)
Lizard, LlZv1: 560 592 KB (94%)
Lizard, FastLZ4 + Huffman: 562 747 KB (95%)
Lizard, LlZv1 + Huffman: 560 040 KB (94%)
My repack choice (command: precompmt+reflate+xpzlib+xcrilayla+lolz, including oodle7 (Selkie) using Knight Compressor v1.0): 530 846 KB (89%)
Xtool FitGirl Repack: 492 634 KB (all files except DirectX) (82%)

NOTE: Solid on, Thread 1, Word Size 273 of 7Z 21.07 and 7Z 21.03 ZS Standard

I tried everything, but it did not work, however I tried using Knight Compressor v1.0, using command of precompmt+reflate+xpzlib+xcrilayla+lolz (including oodlev7, s), but it did well :/ : 530 846 KB.
However, I want to make my repack smaller as/than FitGirl.
Is there any I can make smaller my repack than LZMA?

Thanks,
Unjk223

[KaktoR]

Your chain doesn't make any sense.

I recommmend you find out first what compression is used in input file.

[:( Sad8669]

Using alot of mixed compressors and precompressors won't help achieve a better result, it just gets worse.

[Unjk223]

Oh.... okay, that is weird. How can I avoid worst? I think most repackers using Xtool only, not the Knight Compressor any like that, I looked FitGirl temp files of Hedon Bloodrite.

[KaktoR]

XTool is a precompressor. Simply said, it unpacks files to get better compression with lzma or lolz afterwards. srep between them removes duplicates from the inputs.

[:( Sad8669]

Use xt_zlib+reflate and rip the bonus soundtrack.
I will post the results after 15 minutes.

Hedon.ipk3 | (541 MB) -> (1.98 GB) |

[Unjk223]

Originally Posted by :( Sad8669

Use xt_zlib+reflate and rip the bonus soundtrack.
I will post the results after 15 minutes.

Okay, during your 15 minutes, I just looked up, that FitGirl explained everything about her repack on Reddit: https://reddit.com/r/CrackWatch/comm...anion/gyhkgp5/

Some games FitGirl don't use Xtool like Home Behind 2, which does not say repack tool library by Xtool... like Unity, others game engine will be Xtool.

Regards
- Unjk223

[KaktoR]

Input: Hedon.ipk3 (568.033.789 Bytes)

Method: xtool:zlib+xtool:reflate+srep:m3f+lzma

Compressed 1 file, 568,033,789 => 529,491,980 bytes. Ratio 93.21%
Compression time: cpu 2.13 sec/real 376.67 sec = 1%. Speed 1.51 mB/s
All OK

You probably will get slightly better output with lolz instead of lzma, but it is much more time and cpu resources consuming. Keep that in mind.

If you want you can compress the soundtrack mp3's with srep+mpz

Edit:
Input: Soundtrack mp3's (unpacked zip file)
Method: mpz

Compressed 39 files, 469,898,266 => 369,460,114 bytes. Ratio 78.63%
Compression time: cpu 0.39 sec/real 64.55 sec = 1%. Speed 7.28 mB/s
All OK

[:( Sad8669]

Games come with compression like zlib,oodle etc, xtool expands the streams and srep removes the duplicates, then comes the lolz which finishes things up.

Ofcourse FG explained it more, it doesn't matter if someones explains it, if you spend enough time here and make enough mistakes, you will get it.

PS:- It's gonna take time, 15 minutes was a wrong call.

[kj911]

Unjk223: Trying testing to switches:

xZlib+msc+srep:m3f:a0+lolz
xtool:zlib+xtool:reflate+msc+srep:m3f:a0+lolz

or replace msc to mpz. and add delta before srep. Use lolz 128/256mb dictionary and fba4096, lc1023 and many switches.

[Unjk223]

Originally Posted by KaktoR

Input: Hedon.ipk3 (568.033.789 Bytes)

Method: xtool:zlib+xtool:reflate+srep:m3f+lzma

Compressed 1 file, 568,033,789 => 529,491,980 bytes. Ratio 93.21%
Compression time: cpu 2.13 sec/real 376.67 sec = 1%. Speed 1.51 mB/s
All OK

You probably will get slightly better output with lolz instead of lzma, but it is much more time and cpu resources consuming. Keep that in mind.

If you want you can compress the soundtrack mp3's with srep+mpz

Thanks for information, but I'm gonna test compress only one file of Hedon.ipk3 with my command method: precompmt+reflate+xpzlib+srep:m5f:a64+lolz with Knight Compressor v1.0.

About MP3 soundtracks, this is easy .

I wanna make sure what version are you using of Xtool, so I will compress next time? Example Xtool v0.9.

I make sure post my result, it will take about 0.5-1 hour.

[Unjk223]

Originally Posted by kj911

Unjk223: Trying testing to switches:

xZlib+msc+srep:m3f:a0+lolz
xtool:zlib+xtool:reflate+msc+srep:m3f:a0+lolz

or replace msc to mpz. and add delta before srep. Use lolz 128/256mb dictionary and fba4096, lc1023 and many switches.

I make sure switches compressions possible until I get same repack size of FitGirl repack.

[Unjk223]

Originally Posted by :( Sad8669

Use xt_zlib+reflate and rip the bonus soundtrack.
I will post the results after 15 minutes.

Hedon.ipk3 | (541 MB) -> (1.98 GB) |

366% ratio? I made math calculation to measure your ratio compressed:

1980 MB / 541 MB * 100 = 365,9889094269871

I think compressed file is bigger (1.98 GB) than original file (541 MB)

- Unjk223

[KaktoR]

Originally Posted by Unjk223

I wanna make sure what version are you using of Xtool, so I will compress next time? Example Xtool v0.9.

The newest
https://fileforums.com/showthread.php?t=102832

Originally Posted by Unjk223

366% ratio? I made math calculation to measure your ratio compressed:

1980 MB / 541 MB * 100 = 365,9889094269871

I think compressed file is bigger (1.98 GB) than original file (541 MB)

- Unjk223

Game devs compressed this file with zlib/reflate. Like I said, xtool is a precompressor; it unpacks inputs to compress them better afterwards. Even FitGirl explained it in the link from reddit you posted. This is the main behavior of xtool.

[Unjk223]

Here is what I compressed using Knight Compressor v1.0:

*---------- * your files will be compressed using those settings *----------*
             precompmt+reflate+xpzlib+srep:m5f:a64+lolz
Press any key to continue . . .
  Choose your input file path :
  Your input file path is "G:\HedonIPK3\*"

Press any key to continue . . .
FreeArc 0.67 (December 12 2012) Creating archive: Data_out\Data.cpi using precompmt+reflate+xpzlib+srep:m5f:a64+lolz
Memory for compression 0b, decompression 0b, cache 16mb
Compressing 1 file, 568,033,789 bytes
  Compressing Hedon.ipk3                                                 10%
Compressing 2,055,996,905 bytes with xtool e:precomp:c128m,t4:zlib InFile
 10%
Errorlevel=0
 10%
Compressing 2,114,922,867 bytes with srep -m5f -a64  InFile -
SREP 3.93 beta (September 30, 2014): input size 2016 mb, memory used 656 mb, -m5f -l512 -c256 -a64/16 -ia+ -hash=vmac -b8mb
7%: 150,994,944 -> 123,547,366: 81.82%.  Cpu 177 mb/s (0.812 sec), real 152 mb/s (0.946 sec) = 86%.  Remains 00:12      9%: 209,715,200 -> 182,242,760: 86.90%.  Cpu 194 mb/s (1.031 sec), real 173 mb/s (1.156 sec) = 89%.  Remains 00:10      12%: 268,435,456 -> 240,943,789: 89.76%.  Cpu 202 mb/s (1.266 sec), real 187 mb/s (1.371 sec) = 92%.  Remains 00:09     15%: 318,767,104 -> 291,210,859: 91.36%.  Cpu 199 mb/s (1.531 sec), real 189 mb/s (1.609 sec) = 95%.  Remains 00:09     17%: 369,098,752 -> 341,484,115: 92.52%.  Cpu 205 mb/s (1.719 sec), real 194 mb/s (1.812 sec) = 95%.  Remains 00:08     19%: 411,041,792 -> 383,399,147: 93.27%.  Cpu 199 mb/s (1.969 sec), real 192 mb/s (2.042 sec) = 96%.  Remains 00:08     20%: 436,207,616 -> 408,535,042: 93.66%.  Cpu 189 mb/s (2.203 sec), real 184 mb/s (2.261 sec) = 97%.  Remains 00:09     22%: 478,150,656 -> 450,201,491: 94.15%.  Cpu 186 mb/s (2.453 sec), real 184 mb/s (2.474 sec) = 99%.  Remains 00:08     25%: 545,259,520 -> 517,201,782: 94.85%.  Cpu 192 mb/s (2.703 sec), real 193 mb/s (2.693 sec) = 100%.  Remains 00:08    28%: 612,368,384 -> 584,280,350: 95.41%.  Cpu 197 mb/s (2.969 sec), real 201 mb/s (2.907 sec) = 102%.  Remains 00:07    32%: 679,477,248 -> 647,247,941: 95.26%.  Cpu 201 mb/s (3.219 sec), real 208 mb/s (3.121 sec) = 103%.  Remains 00:07
35%: 746,586,112 -> 714,345,332: 95.68%.  Cpu 205 mb/s (3.469 sec), real 213 mb/s (3.346 sec) = 104%.  Remains 00:06
38%: 813,694,976 -> 780,432,118: 95.91%.  Cpu 209 mb/s (3.719 sec), real 218 mb/s (3.562 sec) = 104%.  Remains 00:06
40%: 855,638,016 -> 819,506,316: 95.78%.  Cpu 207 mb/s (3.938 sec), real 216 mb/s (3.773 sec) = 104%.  Remains 00:05
41%: 880,803,840 -> 841,212,969: 95.51%.  Cpu 202 mb/s (4.156 sec), real 208 mb/s (4.031 sec) = 103%.  Remains 00:06
43%: 914,358,272 -> 865,410,126: 94.65%.  Cpu 197 mb/s (4.422 sec), real 203 mb/s (4.286 sec) = 103%.  Remains 00:06
44%: 939,524,096 -> 888,128,923: 94.53%.  Cpu 192 mb/s (4.672 sec), real 197 mb/s (4.540 sec) = 103%.  Remains 00:06
45%: 964,689,920 -> 911,095,431: 94.44%.  Cpu 188 mb/s (4.891 sec), real 192 mb/s (4.782 sec) = 102%.  Remains 00:06
46%: 989,855,744 -> 932,111,825: 94.17%.  Cpu 184 mb/s (5.141 sec), real 187 mb/s (5.043 sec) = 102%.  Remains 00:06
47%: 1,006,632,960 -> 946,801,922: 94.06%.  Cpu 181 mb/s (5.312 sec), real 182 mb/s (5.266 sec) = 101%.  Remains 00:06
48%: 1,031,798,784 -> 968,855,568: 93.90%.  Cpu 179 mb/s (5.484 sec), real 179 mb/s (5.484 sec) = 100%.  Remains 00:06
49%: 1,048,576,000 -> 979,753,184: 93.44%.  Cpu 175 mb/s (5.703 sec), real 175 mb/s (5.699 sec) = 100%.  Remains 00:06
50%: 1,073,741,824 -> 999,552,373: 93.09%.  Cpu 171 mb/s (6.000 sec), real 171 mb/s (5.979 sec) = 100%.  Remains 00:06
51%: 1,090,519,040 -> 1,007,430,181: 92.38%.  Cpu 169 mb/s (6.172 sec), real 168 mb/s (6.199 sec) = 100%.  Remains 00:06
52%: 1,115,684,864 -> 1,021,888,070: 91.59%.  Cpu 165 mb/s (6.453 sec), real 164 mb/s (6.500 sec) = 99%.  Remains 00:06
53%: 1,140,850,688 -> 1,029,491,142: 90.24%.  Cpu 164 mb/s (6.625 sec), real 161 mb/s (6.754 sec) = 98%.  Remains 00:06
55%: 1,166,016,512 -> 1,042,191,795: 89.38%.  Cpu 163 mb/s (6.828 sec), real 158 mb/s (7.024 sec) = 97%.  Remains 00:06
55%: 1,182,793,728 -> 1,051,812,533: 88.93%.  Cpu 161 mb/s (7.000 sec), real 155 mb/s (7.277 sec) = 96%.  Remains 00:06
57%: 1,207,959,552 -> 1,066,802,848: 88.31%.  Cpu 160 mb/s (7.219 sec), real 152 mb/s (7.593 sec) = 95%.  Remains 00:06
60%: 1,275,068,416 -> 1,071,847,543: 84.06%.  Cpu 165 mb/s (7.391 sec), real 156 mb/s (7.802 sec) = 95%.  Remains 00:05
65%: 1,375,731,712 -> 1,075,196,128: 78.15%.  Cpu 172 mb/s (7.609 sec), real 164 mb/s (8.006 sec) = 95%.  Remains 00:04
71%: 1,509,949,440 -> 1,076,897,284: 71.32%.  Cpu 184 mb/s (7.844 sec), real 175 mb/s (8.209 sec) = 96%.  Remains 00:03
74%: 1,585,446,912 -> 1,085,245,554: 68.45%.  Cpu 188 mb/s (8.062 sec), real 179 mb/s (8.438 sec) = 96%.  Remains 00:03
76%: 1,619,001,344 -> 1,096,709,745: 67.74%.  Cpu 189 mb/s (8.172 sec), real 178 mb/s (8.676 sec) = 94%.  Remains 00:03
78%: 1,652,555,776 -> 1,125,659,114: 68.12%.  Cpu 187 mb/s (8.438 sec), real 176 mb/s (8.945 sec) = 94%.  Remains 00:02
79%: 1,677,721,600 -> 1,149,003,139: 68.49%.  Cpu 186 mb/s (8.625 sec), real 175 mb/s (9.157 sec) = 94%.  Remains 00:02
80%: 1,702,887,424 -> 1,171,953,407: 68.82%.  Cpu 184 mb/s (8.812 sec), real 173 mb/s (9.363 sec) = 94%.  Remains 00:02
82%: 1,736,441,856 -> 1,191,123,000: 68.60%.  Cpu 183 mb/s (9.031 sec), real 172 mb/s (9.649 sec) = 94%.  Remains 00:02
83%: 1,769,996,288 -> 1,209,511,082: 68.33%.  Cpu 183 mb/s (9.203 sec), real 171 mb/s (9.856 sec) = 93%.  Remains 00:02
84%: 1,795,162,112 -> 1,220,793,615: 68.00%.  Cpu 182 mb/s (9.391 sec), real 170 mb/s (10.056 sec) = 93%.  Remains 00:02
86%: 1,820,327,936 -> 1,235,610,955: 67.88%.  Cpu 180 mb/s (9.641 sec), real 167 mb/s (10.377 sec) = 93%.  Remains 00:02
86%: 1,837,105,152 -> 1,238,104,177: 67.39%.  Cpu 179 mb/s (9.812 sec), real 166 mb/s (10.585 sec) = 93%.  Remains 00:02
88%: 1,870,659,584 -> 1,241,836,506: 66.38%.  Cpu 178 mb/s (10.000 sec), real 165 mb/s (10.809 sec) = 93%.  Remains 00:088%: 1,879,048,192 -> 1,245,698,925: 66.29%.  Cpu 177 mb/s (10.141 sec), real 163 mb/s (11.014 sec) = 92%.  Remains 00:090%: 1,920,991,232 -> 1,250,639,148: 65.10%.  Cpu 177 mb/s (10.344 sec), real 163 mb/s (11.239 sec) = 92%.  Remains 00:092%: 1,962,934,272 -> 1,252,670,502: 63.82%.  Cpu 178 mb/s (10.500 sec), real 163 mb/s (11.487 sec) = 91%.  Remains 00:094%: 1,988,100,096 -> 1,254,780,859: 63.11%.  Cpu 178 mb/s (10.656 sec), real 162 mb/s (11.705 sec) = 91%.  Remains 00:095%: 2,030,043,136 -> 1,256,618,908: 61.90%.  Cpu 179 mb/s (10.844 sec), real 162 mb/s (11.917 sec) = 91%.  Remains 00:097%: 2,063,597,568 -> 1,259,732,223: 61.05%.  Cpu 179 mb/s (11.016 sec), real 162 mb/s (12.174 sec) = 90%.  Remains 00:098%: 2,088,763,392 -> 1,262,357,747: 60.44%.  Cpu 179 mb/s (11.109 sec), real 161 mb/s (12.400 sec) = 90%.  Remains 00:099%: 2,113,929,216 -> 1,268,053,406: 59.99%.  Cpu 180 mb/s (11.219 sec), real 160 mb/s (12.601 sec) = 89%.  Remains 00:0100%: 2,114,922,867 -> 1,268,717,250: 59.99%.  Cpu 180 mb/s (11.219 sec), real 160 mb/s (12.608 sec) = 89%.  Remains 00:00
Decompression memory is 47 mb.  306,996 matches = 4,911,936 bytes = 0.39% of file

Errorlevel=0

Compressing 1,268,719,138 bytes with lolz_x64.exe -mc8 -tt4 -mt4 -mtb512 -d128 -fba4096 -dto0 -dm00 -pc2 -bm4 -blr4 -bll8 -blo8 -lm0 -bc4 -cm1 -mc256 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

lolz v22c4b [Dec 30 2018  16:44:58]

Settings:
----------------------------- detection settings -----------------------------
 detection is enabled(-dt)
 use probs propagate(-dtp[0..1]): 1      use brute detect(-dtb[0..1]): 0
 o1 detect(-dto[0..1]): 0
 raw graphic mm detect(-dtm[0..1]): 1    width detect algo(-dtw[0..1]): 1
 dxt detect(-dtd[0..1]): 1
--------------------------- multithreading settings --------------------------
 mt type(-mtt[0..1]): 0
 mt threads(-mt[1..16]): 4               mt block size MB(-mtb[2..512]): 64m
---------------------------- lz & parser settings ----------------------------
 dict size in MB(-d[0..2047]): 128m      num parser arrivals(-tt[1..256]): 4
 parser hard lim(-oh[8..14]): 12         parser soft lim(-os[0..hard_lim]): 8
 fb type A(-fba[0..4096,0=off]): 4096    fb type B(-fbb[0..4096,0=off]): 0
 calc all literals(-al[0..1]): 1         slow parser mode (mlr)(-x[0..2]): 0
 move some of calculations to lz matchfinder thread(-ac[0..1]): 0
---------------------------- matchfinder settings ----------------------------
 use only lz matchfinder (-rt[0..2]: 0)
 lz mf cycles(-mc[2..1023]): 256
 use long distance mf(-ldmf[0..1]): 1
 ldmf min len exp(-ldl[5..12]): 8        ldmf compr/mem coeff (-ldc[0..9]): 1
 ldmf efficient lvl (-lde[0..2]): 0
------------------------------- models settings ------------------------------
general  models :
 use simple cm(-cm[0..1]): 1
 context mixer o1 ctx (-bc[0..6]): 4
 use only o1 nibble literal model (-lm[0..4]): 0
 literal hi o1 ctx (-blo[0..8]): 8       literal lo o1 ctx (-bll[0..8]): 8
 literal hi rep0lit ctx (-blr[0..6]): 4
 match flag rep0lit ctx (-bm[0..6]): 4
 all -p* settings are ignored due to using detect
dxt models :
 dxt color(X) and alpha(Y) model (-dmXY[X[0..3], Y[0..4]]: 00)
raw graphic models :
 color mdl options (-gmXY[X[0..2], Y[0..1]]): 00

100.00%  ldmf dec_mem=0m    0:12:26

thread 0 : w/o manager mem_usage = 0 kb,  manager mem_usage = 0 kb
alloc mem = 8192 kb,  stat compr_size = 21
matches num = 1,  additional mem overhead = 2360 kb
thread 1 : w/o manager mem_usage = 0 kb,  manager mem_usage = 0 kb
alloc mem = 0 kb,  stat compr_size = 10
matches num = 0,  additional mem overhead = 2168 kb
thread 2 : w/o manager mem_usage = 0 kb,  manager mem_usage = 0 kb
alloc mem = 0 kb,  stat compr_size = 10
matches num = 0,  additional mem overhead = 2168 kb
thread 3 : w/o manager mem_usage = 0 kb,  manager mem_usage = 0 kb
alloc mem = 0 kb,  stat compr_size = 10
matches num = 0,  additional mem overhead = 2168 kb

o1 model                 : 1'098'807 kb
raw graphic model 8 bit  : 126'323 kb
raw graphic model 16 bit : 633 kb
raw graphic model 32 bit : 12'673 kb
dxt5 model               : 545 kb
total size               : 1'238'983 kb

total decode mem usage = 799mb

100%
Errorlevel=0

Compressed 1 file, 568,033,789 => 519,041,185 bytes. Ratio 91.3%
Compression time: cpu 10.27 secs, real 1889.67 secs. Speed 301 kB/s
All OK

*******************************************************************************
                                    Finished
*******************************************************************************
 Started   - 18.02.2022 - 22:06:11,18

 Finished  - 18.02.2022 - 22:37:42,17

Press any key to continue . . .

Not bad... but almost close to FitGirl repack size...