Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

 
 
Thread Tools Search this Thread Display Modes
Prev Previous Post   Next Post Next
  #1  
Old Today, 09:30
sajmon83's Avatar
sajmon83 sajmon83 is online now
Registered User
 
Join Date: Jun 2017
Location: Poland
Posts: 48
Thanks: 253
Thanked 58 Times in 35 Posts
sajmon83 is on a distinguished road
ROFZ Compressor

ROFZ – LZ compressor with texture / audio / raw-data models

ROFZ is a lossless compressor for repacking games: optimal-parsing LZ77 + range coder with a per-block literal mixer, plus its own models for the data LZ codes badly. It finds that data anywhere in the stream, by header or by content:
  • BC1–BC7 / DXT textures – DDS, IWI, VTF headers, and BC1/3/4/5 without any header. Textures are chained, and a texture can be coded against an earlier one that shares its blocks.
  • PCM and ADPCM audio (WAV / AIFF, 8/16/24-bit, float, IMA / MS ADPCM) – cascade of sign-LMS predictors.
  • Raw multi-channel data – uncompressed images, vertex / animation arrays (adaptive linear predictor + context mixing).
  • x86 filter, multithreaded compression and decompression, stdin/stdout for FreeArc.
Std-only, Windows x64, needs an AVX2 CPU. Streams decode identically with any thread count.

Neural-network models (CPU only, integer arithmetic; trained offline on samples of 75 games other than Memoriapolis, so as not to skew the results)
  • Online mixing networks – the literal mixer, the BC pixel-index mixer and the raw-data model combine their context predictions with a single-layer logistic network that learns while coding (identically in encoder and decoder).
  • Pre-trained start states – the literal mixer and the raw-data model (weights, context cells, bias, APM) start from states learned on game data instead of even odds (~580 KB of tables in the exe), so small blocks and block starts code smaller.
  • Literal parameter network – a one-layer softmax network over 22 cheap statistics of a block (order-0/1 and position-context entropies, share of zeros, record-stride hints) proposes the two most promising literal-context settings for the encoder's test-coding (trained on 600 blocks from 75 game samples).
  • Texture classifier – a logistic regression (one neuron) over 18 statistics of a texture candidate predicts whether the texture model will beat LZ; rejected candidates skip the trial coding (faster compression).
  • The gains are small (0.02–0.1%) but free for the decoder. -nn turns all trained models off.

Levels
  • -l1 … -l4: fast-decoding models, kept at about 200 MB/s decompression on 16 threads (-ds N changes it). -l1 compresses fastest, -l4 smallest. Default: -l4.
  • -l5 … -l8: full models, smallest output. -l5 compresses fastest, -l8 smallest.
  • -m = memory budget (% of RAM), -t = threads, -ds = decompression speed to keep (MB/s, any level).

Benchmark
Ryzen 7 5800X3D (8C/16T), 64 GB RAM, NVMe SSD, Windows 11. Command line, one run after another on an idle PC, all 16 threads; every output was decompressed and compared byte for byte with the input. MB/s refer to the compressor's input.

Settings
  • LOLZ ULTRA (DiskSpan's LOLZ_ULTRA alias): lolz -dtb1 -d263 -mtt1 -mt16 -mc1023, decoded with CLS-LOLZ_D_x64
  • TARZ 1.0.3 at its best (LOLZ-ULTRA-like settings): tarz c -t16 -d256 -mtt1 -tt16 -mc1023 -ow4096 -fb256, tarz d -t16
  • ROFZ 0.17.0 (the benchmark was made with 0.17.0): rofz c -lN -m85%, rofz d (defaults otherwise)

Inputs and precompression
  • Per-type sets: 37 files of up to 16 MB cut from game data samples (files of one kind put together): PCM audio, IMA / MS ADPCM audio, BC1–BC7 / DXT textures (DDS, VTF, IWI, XBM and header-less), raw images (BMP / TGA / RAW / TIF), text, Unity / CR2W / Source structures, executables, meshes, animations, BSP maps, PAK / VPK archives. No precompression – the same file for every compressor.
  • Memoriapolis (Unity 2022.3, 7 832 files, 7 267 959 185 B): files in Arc order (by extension, then path); the LZ4 / LZ4HC blocks of the 4 713 UnityFS archives restored to raw data, only blocks that recompress bit for bit) → 9 119 658 087 B; then srep 4.90g-wip -hash- -m3f -a0/0 → 5 355 251 639 B, the input of every run.

Audio: PCM and ADPCM differ. The −21% of the PCM audio sets holds for uncompressed samples only (WAV / AIFF, 8/16/24-bit, float). ADPCM has its own model, but it decodes at ~6 MB/s per thread, so -l1 … -l4 (the default) leave it to LZ and land at about. Therefore, use -l5 … -l8 or -ds 0.

Code:
PCM audio (WAV / AIFF; uncompressed samples): 67 108 864 B input
compressor   output (B)   ratio  vs LOLZ  compression   MB/s  decompression  MB/s
LOLZ ULTRA   39 127 417  58.30%        -       50.4 s   1.33         1.62 s    42
TARZ (best)  40 183 818  59.88%   +2.70%      191.2 s   0.35         2.70 s    25
rofz -l1     30 841 176  45.96%  -21.18%        4.2 s  15.81         0.85 s    79
rofz -l2     30 841 153  45.96%  -21.18%        4.5 s  15.01         0.81 s    82
rofz -l3     30 841 117  45.96%  -21.18%        4.7 s  14.34         0.80 s    84
rofz -l4     30 841 062  45.96%  -21.18%        4.6 s  14.57         0.83 s    81
rofz -l5     30 841 056  45.96%  -21.18%        4.6 s  14.54         0.81 s    83
rofz -l6     30 839 955  45.96%  -21.18%        6.8 s   9.91         0.83 s    81
rofz -l7     30 840 239  45.96%  -21.18%        6.8 s   9.84         0.81 s    83
rofz -l8     30 840 254  45.96%  -21.18%        6.9 s   9.76         0.83 s    81

ADPCM audio (IMA / MS ADPCM WAV): 23 595 328 B input
compressor   output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA   14 637 375  62.04%        -       19.4 s  1.22         0.60 s    39
TARZ (best)  14 557 305  61.70%   -0.55%       94.1 s  0.25         0.73 s    32
rofz -l1     14 841 614  62.90%   +1.40%        6.5 s  3.63         0.40 s    59
rofz -l2     14 655 579  62.11%   +0.12%        9.0 s  2.62         0.44 s    54
rofz -l3     14 767 282  62.59%   +0.89%       12.6 s  1.88         0.42 s    57
rofz -l4     14 715 031  62.36%   +0.53%       14.9 s  1.58         0.42 s    56
rofz -l5     13 144 443  55.71%  -10.20%        7.1 s  3.31         0.50 s    47
rofz -l6     13 114 475  55.58%  -10.40%       10.9 s  2.17         0.51 s    46
rofz -l7     13 111 983  55.57%  -10.42%       13.8 s  1.70         0.53 s    45
rofz -l8     13 109 901  55.56%  -10.44%       16.9 s  1.39         0.53 s    45

textures (DDS / BC1-BC7 / VTF / IWI / XBM): 150 994 944 B input
compressor   output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA   65 330 035  43.27%        -      156.3 s  0.97         3.64 s    41
TARZ (best)  65 011 249  43.06%   -0.49%      285.7 s  0.53         6.87 s    22
rofz -l1     69 948 641  46.33%   +7.07%       51.9 s  2.91         1.72 s    88
rofz -l2     69 036 769  45.72%   +5.67%       61.5 s  2.46         1.74 s    87
rofz -l3     68 975 602  45.68%   +5.58%       75.9 s  1.99         1.74 s    87
rofz -l4     68 830 512  45.58%   +5.36%       88.5 s  1.71         1.73 s    87
rofz -l5     61 524 633  40.75%   -5.82%       69.2 s  2.18         8.54 s    18
rofz -l6     61 091 679  40.46%   -6.49%       95.4 s  1.58         8.93 s    17
rofz -l7     61 030 275  40.42%   -6.58%      110.9 s  1.36         9.27 s    16
rofz -l8     61 004 049  40.40%   -6.62%      135.2 s  1.12         9.09 s    17

raw images (BMP / TGA / RAW / TIF): 83 886 080 B input
compressor   output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA   21 323 178  25.42%        -       78.9 s  1.06         1.60 s    52
TARZ (best)  20 755 681  24.74%   -2.66%      437.9 s  0.19         2.56 s    33
rofz -l1     24 610 417  29.34%  +15.42%       21.0 s  3.99         0.70 s   120
rofz -l2     24 464 511  29.16%  +14.73%       26.6 s  3.15         0.70 s   119
rofz -l3     24 362 914  29.04%  +14.26%       37.0 s  2.27         0.72 s   117
rofz -l4     24 280 411  28.94%  +13.87%       50.0 s  1.68         0.71 s   118
rofz -l5     20 374 255  24.29%   -4.45%       35.5 s  2.36         3.01 s    28
rofz -l6     20 180 470  24.06%   -5.36%       48.3 s  1.74         3.24 s    26
rofz -l7     20 150 628  24.02%   -5.50%       62.2 s  1.35         3.13 s    27
rofz -l8     20 118 927  23.98%   -5.65%       90.3 s  0.93         3.24 s    26

text, structures, executables, meshes, maps: 266 556 789 B input
compressor    output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s
LOLZ ULTRA    95 836 113  35.95%        -      267.1 s  1.00         3.27 s    82
TARZ (best)   95 749 915  35.92%   -0.09%     1301.8 s  0.20         4.68 s    57
rofz -l1     103 232 176  38.73%   +7.72%       65.8 s  4.05         2.39 s   111
rofz -l2     102 161 297  38.33%   +6.60%       92.7 s  2.87         2.53 s   105
rofz -l3     101 522 432  38.09%   +5.93%      133.0 s  2.00         2.56 s   104
rofz -l4     100 950 059  37.87%   +5.34%      164.5 s  1.62         2.66 s   100
rofz -l5      99 296 586  37.25%   +3.61%      147.1 s  1.81         5.42 s    49
rofz -l6      94 025 908  35.27%   -1.89%      283.4 s  0.94         9.01 s    30
rofz -l7      93 433 928  35.05%   -2.51%      417.0 s  0.64         9.24 s    29
rofz -l8      93 169 190  34.95%   -2.78%      673.5 s  0.40         9.19 s    29

Memoriapolis, whole game: 5 355 251 639 B input
compressor      output (B)   ratio  vs LOLZ  compression  MB/s  decompression  MB/s  RAM to decompress
LOLZ ULTRA   1 758 818 797  32.84%        -     1615.1 s  3.32        12.00 s   446           5 698 MB
TARZ (best)  1 772 775 068  33.10%   +0.79%     5779.4 s  0.93        20.30 s   264           2 784 MB
rofz -l1     1 858 615 328  34.71%   +5.67%      564.5 s  9.49        24.68 s   217           4 103 MB
rofz -l4     1 801 250 333  33.64%   +2.41%     1302.6 s  4.11        27.01 s   198           4 081 MB
rofz -l5     1 763 133 782  32.92%   +0.25%     1442.6 s  3.71        54.31 s    99           6 377 MB
rofz -l8     1 739 267 961  32.48%   -1.11%     3545.8 s  1.51        82.51 s    65           8 847 MB
Per set (output bytes)
Code:
set                 LOLZ ULTRA  TARZ (best)     rofz -l1     rofz -l2     rofz -l3     rofz -l4     rofz -l5     rofz -l6     rofz -l7     rofz -l8
lit/text             1 939 745    1 920 604    2 262 808    2 227 316    2 218 327    2 215 268    2 200 847    1 862 894    1 862 845    1 862 066
lit/text_utf16         782 807      768 166      888 579      850 714      818 020      800 900      800 900      732 423      731 687      731 892
lit/struct_unity     4 622 416    4 613 608    4 849 278    4 751 920    4 734 144    4 691 595    4 620 495    4 559 034    4 543 832    4 522 125
lit/struct_cr2w     12 435 801   12 466 536   12 639 172   12 587 096   12 569 746   12 555 434   12 482 358   12 407 004   12 395 590   12 388 942
lit/struct_misc      1 646 195    1 642 642    2 542 813    2 416 790    2 414 836    2 386 836    2 385 877    1 775 473    1 767 250    1 750 461
lit/struct_source   12 877 500   12 860 749   12 995 671   12 980 473   12 903 204   12 852 635   12 642 857   12 426 081   12 393 627   12 373 962
lit/exe64            4 402 081    4 000 834    4 405 168    4 347 824    4 301 554    4 267 053    4 132 456    3 996 796    3 981 898    3 968 525
lit/exe32            9 230 031    9 100 271   11 300 627   11 260 205   11 248 602   11 237 582   11 187 425    9 041 833    9 036 342    9 035 011
lit/exe_net          3 619 668    3 589 282    3 814 151    3 774 163    3 726 540    3 703 505    3 646 090    3 567 644    3 557 735    3 549 889
lit/mesh             7 019 892    7 061 608    8 106 947    8 067 894    8 024 947    8 005 355    7 929 739    7 017 507    6 983 564    6 976 781
lit/anim             5 093 373    5 523 928    5 654 300    5 594 216    5 457 922    5 407 425    5 162 291    4 924 904    4 831 283    4 832 529
lit/map_bsp          2 044 624    2 073 579    2 928 230    2 813 633    2 737 254    2 656 450    2 457 677    2 411 648    2 134 791    2 011 830
lit/terrain_nav      5 181 137    5 001 339    5 356 031    5 261 473    5 188 097    5 139 100    5 010 374    4 977 348    4 948 536    4 926 081
lit/font             3 279 030    3 265 130    3 537 714    3 426 914    3 419 834    3 308 573    3 148 620    3 008 360    2 960 486    2 944 509
lit/shader             294 593      285 262      312 772      298 792      292 592      288 823      288 823      287 478      287 627      284 382
lit/pack_pak         9 394 270    9 534 756    9 561 099    9 531 863    9 516 861    9 508 237    9 450 969    9 316 653    9 312 523    9 310 209
lit/pack_vpk        11 972 950   12 041 621   12 076 816   11 970 011   11 949 952   11 925 288   11 748 788   11 712 828   11 704 312   11 699 996
audio/audio_pcm16   12 621 005   12 980 497    9 844 657    9 844 649    9 844 628    9 844 577    9 844 575    9 843 976    9 844 260    9 844 276
audio/WAV16          9 020 554    9 567 334    7 417 279    7 417 272    7 417 259    7 417 258    7 417 257    7 417 068    7 417 068    7 417 068
audio/WAV24          8 424 908    8 093 706    6 161 722    6 161 720    6 161 714    6 161 711    6 161 710    6 161 510    6 161 510    6 161 510
audio/AIFF           9 060 950    9 542 281    7 417 518    7 417 512    7 417 516    7 417 516    7 417 514    7 417 401    7 417 401    7 417 400
raw/img_raw          4 244 729    4 129 180    4 327 272    4 299 776    4 257 292    4 219 785    4 094 807    3 946 790    3 925 761    3 905 797
raw/BMP              4 286 406    4 176 200    5 026 948    5 005 881    4 994 612    4 979 539    4 096 476    4 076 462    4 072 811    4 067 539
raw/TGA              4 283 867    4 175 178    5 027 335    5 005 923    4 994 678    4 979 656    4 096 535    4 078 837    4 072 918    4 067 784
raw/RAW              4 254 850    4 126 583    5 319 411    5 244 519    5 226 319    5 220 814    4 243 943    4 236 600    4 237 186    4 235 932
raw/TIF              4 253 326    4 148 540    4 909 451    4 908 412    4 890 013    4 880 617    3 842 494    3 841 781    3 841 952    3 841 875
tex/tex_bc1          8 534 089    8 451 644    9 049 159    8 999 048    9 013 355    9 003 513    8 231 756    8 226 511    8 224 326    8 223 206
tex/tex_bc3          4 775 742    4 778 892    5 030 075    5 029 991    5 029 618    5 029 376    4 553 433    4 552 990    4 552 975    4 552 907
tex/tex_bc45         4 987 145    4 979 232    5 524 199    5 376 167    5 346 919    5 327 043    5 185 965    5 075 383    5 042 839    5 031 738
tex/tex_bc7         11 390 057   11 140 226   11 874 864   11 622 766   11 631 007   11 603 221   10 979 436   10 869 072   10 865 053   10 862 826
tex/tex_bc6         12 295 850   12 353 859   13 307 248   13 040 088   13 059 948   13 028 860    9 160 955    9 160 864    9 160 864    9 160 862
tex/dds              1 265 829    1 200 701    1 250 563    1 217 712    1 194 798    1 182 005    1 133 061    1 133 020    1 133 020    1 133 020
tex/tex_vtf          4 831 058    4 865 011    5 163 232    5 110 175    5 100 603    5 090 718    4 861 782    4 855 095    4 854 611    4 853 849
tex/tex_xbm          9 254 180    9 334 486   10 049 317    9 940 778    9 906 648    9 877 155    9 421 032    9 230 276    9 222 629    9 219 057
tex/tex_iwi          7 996 085    7 907 198    8 699 984    8 700 044    8 692 706    8 688 621    7 997 213    7 988 468    7 973 958    7 966 584
adpcm/ADPCM_IMA      7 954 668    7 967 607    8 012 745    7 952 591    7 942 684    7 916 485    7 085 261    7 066 231    7 062 727    7 061 197
adpcm/ADPCM_MS       6 682 707    6 589 698    6 828 869    6 702 988    6 824 598    6 798 546    6 059 182    6 048 244    6 049 256    6 048 704
FreeArc / DiskSpan (arc.ini):
Code:
[External compressor:rofz]
header    = 0
packcmd   = "rofz.exe" c {options} - - <stdin> <stdout>
unpackcmd = "rofz.exe" d - - <stdin> <stdout>
Use e.g. -mrofz:l4:m65% after your precompressors. Presets: rofz:l2:m45% (fast), rofz:l4:m65% (normal), rofz:l7:m85% (high), rofz:l8:m85% (ultra). In an installer only the unpackcmd is needed (d -t100p - - <stdin> <stdout>).

Changes
  • 0.17.1 (stream format v17: 0.17.0 streams decode; streams made with -nw need 0.17.1)
    • switches for the single models and the trained models.
    • default level 4 (was 3)
    • fix: audio from a WAV / ADPCM header that claims more data than follows it (a cut or broken file) ran on over the next data and coded it as audio. Textures and images found by their headers are now taken out of the audio, and the last 32 KB of every audio stream are coded by the audio model and by LZ; where LZ wins, a binary search in 64 KB steps finds where the stream turns into other data and the audio ends there (cut-WAV test files: −2.8% at -l4, −15.5% at -l7). Normal audio codes as before.
  • 0.17.0 first release

Download: rofz-0.17.1-win64.7z (rofz.exe + README)
SHA-256: 915342c2ebde3636d0ca644c7c0e7557e4b7bc4aa70ddb4161 65b6abe8f3157e (779 321 B)

Feedback and results on other games are welcome, especially where it loses. And 200MB of samples
Attached Files
File Type: 7z rofz-0.17.0-win64.7z (752.2 KB, 7 views)
File Type: 7z rofz-0.17.1-win64.7z (761.1 KB, 5 views)

Last edited by sajmon83; Today at 12:10.
Reply With Quote
The Following 2 Users Say Thank You to sajmon83 For This Useful Post:
DomoVoi_96 (Today), Masquerade (Today)
Sponsored Links
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New Fox Kompressor 1.01c Ultimate felice2011 Conversion Tutorials 31 07-06-2023 15:58
Ultimate Conversion Compressor (UCC) vollachr Conversion Tutorials 55 26-04-2021 10:27
Kitty File Compressor (STDIO Patch) 78372 Conversion Tutorials 1 31-10-2019 08:46
RAYcompressor - the ultimate AIO compressor rayated Conversion Tutorials 38 11-04-2018 01:10
Tiny Compressor 78372 Conversion Tutorials 1 24-11-2017 14:27



All times are GMT -7. The time now is 14:30.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2026, vBulletin Solutions Inc.
FileForums @ https://fileforums.com