| sajmon83 |
22-11-2026 09:30 |
ROFZ Compressor
2 Attachment(s)
ROFZ – LZ compressor with texture / audio / raw-data models
ROFZ is a lossless compressor for repacking games: optimal-parsing LZ77 + range coder with a per-block literal mixer, plus its own models for the data LZ codes badly. It finds that data anywhere in the stream, by header or by content: - BC1–BC7 / DXT textures – DDS, IWI, VTF headers, and BC1/3/4/5 without any header. Textures are chained, and a texture can be coded against an earlier one that shares its blocks.
- PCM and ADPCM audio (WAV / AIFF, 8/16/24-bit, float, IMA / MS ADPCM) – cascade of sign-LMS predictors.
- Raw multi-channel data – uncompressed images, vertex / animation arrays (adaptive linear predictor + context mixing).
- x86 filter, multithreaded compression and decompression, stdin/stdout for FreeArc.
Std-only, Windows x64, needs an AVX2 CPU. Streams decode identically with any thread count.
Neural-network models (CPU only, integer arithmetic; trained offline on samples of 75 games other than Memoriapolis, so as not to skew the results) - Online mixing networks – the literal mixer, the BC pixel-index mixer and the raw-data model combine their context predictions with a single-layer logistic network that learns while coding (identically in encoder and decoder).
- Pre-trained start states – the literal mixer and the raw-data model (weights, context cells, bias, APM) start from states learned on game data instead of even odds (~580 KB of tables in the exe), so small blocks and block starts code smaller.
- Literal parameter network – a one-layer softmax network over 22 cheap statistics of a block (order-0/1 and position-context entropies, share of zeros, record-stride hints) proposes the two most promising literal-context settings for the encoder's test-coding (trained on 600 blocks from 75 game samples).
- Texture classifier – a logistic regression (one neuron) over 18 statistics of a texture candidate predicts whether the texture model will beat LZ; rejected candidates skip the trial coding (faster compression).
- The gains are small (0.02–0.1%) but free for the decoder. -nn turns all trained models off.
Levels- -l1 … -l4: fast-decoding models, kept at about 200 MB/s decompression on 16 threads (-ds N changes it). -l1 compresses fastest, -l4 smallest. Default: -l4.
- -l5 … -l8: full models, smallest output. -l5 compresses fastest, -l8 smallest.
- -m = memory budget (% of RAM), -t = threads, -ds = decompression speed to keep (MB/s, any level).
Benchmark
Ryzen 7 5800X3D (8C/16T), 64 GB RAM, NVMe SSD, Windows 11. Command line, one run after another on an idle PC, all 16 threads; every output was decompressed and compared byte for byte with the input. MB/s refer to the compressor's input.
Settings- LOLZ ULTRA (DiskSpan's LOLZ_ULTRA alias): lolz -dtb1 -d263 -mtt1 -mt16 -mc1023, decoded with CLS-LOLZ_D_x64
- TARZ 1.0.3 at its best (LOLZ-ULTRA-like settings): tarz c -t16 -d256 -mtt1 -tt16 -mc1023 -ow4096 -fb256, tarz d -t16
- ROFZ 0.17.0 (the benchmark was made with 0.17.0): rofz c -lN -m85%, rofz d (defaults otherwise)
Inputs and precompression- Per-type sets: 37 files of up to 16 MB cut from game data samples (files of one kind put together): PCM audio, IMA / MS ADPCM audio, BC1–BC7 / DXT textures (DDS, VTF, IWI, XBM and header-less), raw images (BMP / TGA / RAW / TIF), text, Unity / CR2W / Source structures, executables, meshes, animations, BSP maps, PAK / VPK archives. No precompression – the same file for every compressor.
- Memoriapolis (Unity 2022.3, 7 832 files, 7 267 959 185 B): files in Arc order (by extension, then path); the LZ4 / LZ4HC blocks of the 4 713 UnityFS archives restored to raw data, only blocks that recompress bit for bit) → 9 119 658 087 B; then srep 4.90g-wip -hash- -m3f -a0/0 → 5 355 251 639 B, the input of every run.
Audio: PCM and ADPCM differ. The −21% of the PCM audio sets holds for uncompressed samples only (WAV / AIFF, 8/16/24-bit, float). ADPCM has its own model, but it decodes at ~6 MB/s per thread, so -l1 … -l4 (the default) leave it to LZ and land at about. Therefore, use -l5 … -l8 or -ds 0.
Code:
PCM audio (WAV / AIFF; uncompressed samples): 67 108 864 B input
compressor output (B) ratio vs LOLZ compression MB/s decompression MB/s
LOLZ ULTRA 39 127 417 58.30% - 50.4 s 1.33 1.62 s 42
TARZ (best) 40 183 818 59.88% +2.70% 191.2 s 0.35 2.70 s 25
rofz -l1 30 841 176 45.96% -21.18% 4.2 s 15.81 0.85 s 79
rofz -l2 30 841 153 45.96% -21.18% 4.5 s 15.01 0.81 s 82
rofz -l3 30 841 117 45.96% -21.18% 4.7 s 14.34 0.80 s 84
rofz -l4 30 841 062 45.96% -21.18% 4.6 s 14.57 0.83 s 81
rofz -l5 30 841 056 45.96% -21.18% 4.6 s 14.54 0.81 s 83
rofz -l6 30 839 955 45.96% -21.18% 6.8 s 9.91 0.83 s 81
rofz -l7 30 840 239 45.96% -21.18% 6.8 s 9.84 0.81 s 83
rofz -l8 30 840 254 45.96% -21.18% 6.9 s 9.76 0.83 s 81
ADPCM audio (IMA / MS ADPCM WAV): 23 595 328 B input
compressor output (B) ratio vs LOLZ compression MB/s decompression MB/s
LOLZ ULTRA 14 637 375 62.04% - 19.4 s 1.22 0.60 s 39
TARZ (best) 14 557 305 61.70% -0.55% 94.1 s 0.25 0.73 s 32
rofz -l1 14 841 614 62.90% +1.40% 6.5 s 3.63 0.40 s 59
rofz -l2 14 655 579 62.11% +0.12% 9.0 s 2.62 0.44 s 54
rofz -l3 14 767 282 62.59% +0.89% 12.6 s 1.88 0.42 s 57
rofz -l4 14 715 031 62.36% +0.53% 14.9 s 1.58 0.42 s 56
rofz -l5 13 144 443 55.71% -10.20% 7.1 s 3.31 0.50 s 47
rofz -l6 13 114 475 55.58% -10.40% 10.9 s 2.17 0.51 s 46
rofz -l7 13 111 983 55.57% -10.42% 13.8 s 1.70 0.53 s 45
rofz -l8 13 109 901 55.56% -10.44% 16.9 s 1.39 0.53 s 45
textures (DDS / BC1-BC7 / VTF / IWI / XBM): 150 994 944 B input
compressor output (B) ratio vs LOLZ compression MB/s decompression MB/s
LOLZ ULTRA 65 330 035 43.27% - 156.3 s 0.97 3.64 s 41
TARZ (best) 65 011 249 43.06% -0.49% 285.7 s 0.53 6.87 s 22
rofz -l1 69 948 641 46.33% +7.07% 51.9 s 2.91 1.72 s 88
rofz -l2 69 036 769 45.72% +5.67% 61.5 s 2.46 1.74 s 87
rofz -l3 68 975 602 45.68% +5.58% 75.9 s 1.99 1.74 s 87
rofz -l4 68 830 512 45.58% +5.36% 88.5 s 1.71 1.73 s 87
rofz -l5 61 524 633 40.75% -5.82% 69.2 s 2.18 8.54 s 18
rofz -l6 61 091 679 40.46% -6.49% 95.4 s 1.58 8.93 s 17
rofz -l7 61 030 275 40.42% -6.58% 110.9 s 1.36 9.27 s 16
rofz -l8 61 004 049 40.40% -6.62% 135.2 s 1.12 9.09 s 17
raw images (BMP / TGA / RAW / TIF): 83 886 080 B input
compressor output (B) ratio vs LOLZ compression MB/s decompression MB/s
LOLZ ULTRA 21 323 178 25.42% - 78.9 s 1.06 1.60 s 52
TARZ (best) 20 755 681 24.74% -2.66% 437.9 s 0.19 2.56 s 33
rofz -l1 24 610 417 29.34% +15.42% 21.0 s 3.99 0.70 s 120
rofz -l2 24 464 511 29.16% +14.73% 26.6 s 3.15 0.70 s 119
rofz -l3 24 362 914 29.04% +14.26% 37.0 s 2.27 0.72 s 117
rofz -l4 24 280 411 28.94% +13.87% 50.0 s 1.68 0.71 s 118
rofz -l5 20 374 255 24.29% -4.45% 35.5 s 2.36 3.01 s 28
rofz -l6 20 180 470 24.06% -5.36% 48.3 s 1.74 3.24 s 26
rofz -l7 20 150 628 24.02% -5.50% 62.2 s 1.35 3.13 s 27
rofz -l8 20 118 927 23.98% -5.65% 90.3 s 0.93 3.24 s 26
text, structures, executables, meshes, maps: 266 556 789 B input
compressor output (B) ratio vs LOLZ compression MB/s decompression MB/s
LOLZ ULTRA 95 836 113 35.95% - 267.1 s 1.00 3.27 s 82
TARZ (best) 95 749 915 35.92% -0.09% 1301.8 s 0.20 4.68 s 57
rofz -l1 103 232 176 38.73% +7.72% 65.8 s 4.05 2.39 s 111
rofz -l2 102 161 297 38.33% +6.60% 92.7 s 2.87 2.53 s 105
rofz -l3 101 522 432 38.09% +5.93% 133.0 s 2.00 2.56 s 104
rofz -l4 100 950 059 37.87% +5.34% 164.5 s 1.62 2.66 s 100
rofz -l5 99 296 586 37.25% +3.61% 147.1 s 1.81 5.42 s 49
rofz -l6 94 025 908 35.27% -1.89% 283.4 s 0.94 9.01 s 30
rofz -l7 93 433 928 35.05% -2.51% 417.0 s 0.64 9.24 s 29
rofz -l8 93 169 190 34.95% -2.78% 673.5 s 0.40 9.19 s 29
Memoriapolis, whole game: 5 355 251 639 B input
compressor output (B) ratio vs LOLZ compression MB/s decompression MB/s RAM to decompress
LOLZ ULTRA 1 758 818 797 32.84% - 1615.1 s 3.32 12.00 s 446 5 698 MB
TARZ (best) 1 772 775 068 33.10% +0.79% 5779.4 s 0.93 20.30 s 264 2 784 MB
rofz -l1 1 858 615 328 34.71% +5.67% 564.5 s 9.49 24.68 s 217 4 103 MB
rofz -l4 1 801 250 333 33.64% +2.41% 1302.6 s 4.11 27.01 s 198 4 081 MB
rofz -l5 1 763 133 782 32.92% +0.25% 1442.6 s 3.71 54.31 s 99 6 377 MB
rofz -l8 1 739 267 961 32.48% -1.11% 3545.8 s 1.51 82.51 s 65 8 847 MB
Per set (output bytes)
Code:
set LOLZ ULTRA TARZ (best) rofz -l1 rofz -l2 rofz -l3 rofz -l4 rofz -l5 rofz -l6 rofz -l7 rofz -l8
lit/text 1 939 745 1 920 604 2 262 808 2 227 316 2 218 327 2 215 268 2 200 847 1 862 894 1 862 845 1 862 066
lit/text_utf16 782 807 768 166 888 579 850 714 818 020 800 900 800 900 732 423 731 687 731 892
lit/struct_unity 4 622 416 4 613 608 4 849 278 4 751 920 4 734 144 4 691 595 4 620 495 4 559 034 4 543 832 4 522 125
lit/struct_cr2w 12 435 801 12 466 536 12 639 172 12 587 096 12 569 746 12 555 434 12 482 358 12 407 004 12 395 590 12 388 942
lit/struct_misc 1 646 195 1 642 642 2 542 813 2 416 790 2 414 836 2 386 836 2 385 877 1 775 473 1 767 250 1 750 461
lit/struct_source 12 877 500 12 860 749 12 995 671 12 980 473 12 903 204 12 852 635 12 642 857 12 426 081 12 393 627 12 373 962
lit/exe64 4 402 081 4 000 834 4 405 168 4 347 824 4 301 554 4 267 053 4 132 456 3 996 796 3 981 898 3 968 525
lit/exe32 9 230 031 9 100 271 11 300 627 11 260 205 11 248 602 11 237 582 11 187 425 9 041 833 9 036 342 9 035 011
lit/exe_net 3 619 668 3 589 282 3 814 151 3 774 163 3 726 540 3 703 505 3 646 090 3 567 644 3 557 735 3 549 889
lit/mesh 7 019 892 7 061 608 8 106 947 8 067 894 8 024 947 8 005 355 7 929 739 7 017 507 6 983 564 6 976 781
lit/anim 5 093 373 5 523 928 5 654 300 5 594 216 5 457 922 5 407 425 5 162 291 4 924 904 4 831 283 4 832 529
lit/map_bsp 2 044 624 2 073 579 2 928 230 2 813 633 2 737 254 2 656 450 2 457 677 2 411 648 2 134 791 2 011 830
lit/terrain_nav 5 181 137 5 001 339 5 356 031 5 261 473 5 188 097 5 139 100 5 010 374 4 977 348 4 948 536 4 926 081
lit/font 3 279 030 3 265 130 3 537 714 3 426 914 3 419 834 3 308 573 3 148 620 3 008 360 2 960 486 2 944 509
lit/shader 294 593 285 262 312 772 298 792 292 592 288 823 288 823 287 478 287 627 284 382
lit/pack_pak 9 394 270 9 534 756 9 561 099 9 531 863 9 516 861 9 508 237 9 450 969 9 316 653 9 312 523 9 310 209
lit/pack_vpk 11 972 950 12 041 621 12 076 816 11 970 011 11 949 952 11 925 288 11 748 788 11 712 828 11 704 312 11 699 996
audio/audio_pcm16 12 621 005 12 980 497 9 844 657 9 844 649 9 844 628 9 844 577 9 844 575 9 843 976 9 844 260 9 844 276
audio/WAV16 9 020 554 9 567 334 7 417 279 7 417 272 7 417 259 7 417 258 7 417 257 7 417 068 7 417 068 7 417 068
audio/WAV24 8 424 908 8 093 706 6 161 722 6 161 720 6 161 714 6 161 711 6 161 710 6 161 510 6 161 510 6 161 510
audio/AIFF 9 060 950 9 542 281 7 417 518 7 417 512 7 417 516 7 417 516 7 417 514 7 417 401 7 417 401 7 417 400
raw/img_raw 4 244 729 4 129 180 4 327 272 4 299 776 4 257 292 4 219 785 4 094 807 3 946 790 3 925 761 3 905 797
raw/BMP 4 286 406 4 176 200 5 026 948 5 005 881 4 994 612 4 979 539 4 096 476 4 076 462 4 072 811 4 067 539
raw/TGA 4 283 867 4 175 178 5 027 335 5 005 923 4 994 678 4 979 656 4 096 535 4 078 837 4 072 918 4 067 784
raw/RAW 4 254 850 4 126 583 5 319 411 5 244 519 5 226 319 5 220 814 4 243 943 4 236 600 4 237 186 4 235 932
raw/TIF 4 253 326 4 148 540 4 909 451 4 908 412 4 890 013 4 880 617 3 842 494 3 841 781 3 841 952 3 841 875
tex/tex_bc1 8 534 089 8 451 644 9 049 159 8 999 048 9 013 355 9 003 513 8 231 756 8 226 511 8 224 326 8 223 206
tex/tex_bc3 4 775 742 4 778 892 5 030 075 5 029 991 5 029 618 5 029 376 4 553 433 4 552 990 4 552 975 4 552 907
tex/tex_bc45 4 987 145 4 979 232 5 524 199 5 376 167 5 346 919 5 327 043 5 185 965 5 075 383 5 042 839 5 031 738
tex/tex_bc7 11 390 057 11 140 226 11 874 864 11 622 766 11 631 007 11 603 221 10 979 436 10 869 072 10 865 053 10 862 826
tex/tex_bc6 12 295 850 12 353 859 13 307 248 13 040 088 13 059 948 13 028 860 9 160 955 9 160 864 9 160 864 9 160 862
tex/dds 1 265 829 1 200 701 1 250 563 1 217 712 1 194 798 1 182 005 1 133 061 1 133 020 1 133 020 1 133 020
tex/tex_vtf 4 831 058 4 865 011 5 163 232 5 110 175 5 100 603 5 090 718 4 861 782 4 855 095 4 854 611 4 853 849
tex/tex_xbm 9 254 180 9 334 486 10 049 317 9 940 778 9 906 648 9 877 155 9 421 032 9 230 276 9 222 629 9 219 057
tex/tex_iwi 7 996 085 7 907 198 8 699 984 8 700 044 8 692 706 8 688 621 7 997 213 7 988 468 7 973 958 7 966 584
adpcm/ADPCM_IMA 7 954 668 7 967 607 8 012 745 7 952 591 7 942 684 7 916 485 7 085 261 7 066 231 7 062 727 7 061 197
adpcm/ADPCM_MS 6 682 707 6 589 698 6 828 869 6 702 988 6 824 598 6 798 546 6 059 182 6 048 244 6 049 256 6 048 704
FreeArc / DiskSpan (arc.ini):
Code:
[External compressor:rofz]
header = 0
packcmd = "rofz.exe" c {options} - - <stdin> <stdout>
unpackcmd = "rofz.exe" d - - <stdin> <stdout>
Use e.g. -mrofz:l4:m65% after your precompressors. Presets: rofz:l2:m45% (fast), rofz:l4:m65% (normal), rofz:l7:m85% (high), rofz:l8:m85% (ultra). In an installer only the unpackcmd is needed ( d -t100p - - <stdin> <stdout>).
Changes- 0.17.1 (stream format v17: 0.17.0 streams decode; streams made with -nw need 0.17.1)
- switches for the single models and the trained models.
- default level 4 (was 3)
- fix: audio from a WAV / ADPCM header that claims more data than follows it (a cut or broken file) ran on over the next data and coded it as audio. Textures and images found by their headers are now taken out of the audio, and the last 32 KB of every audio stream are coded by the audio model and by LZ; where LZ wins, a binary search in 64 KB steps finds where the stream turns into other data and the audio ends there (cut-WAV test files: −2.8% at -l4, −15.5% at -l7). Normal audio codes as before.
- 0.17.0 first release
Download: rofz-0.17.1-win64.7z (rofz.exe + README)
SHA-256: 915342c2ebde3636d0ca644c7c0e7557e4b7bc4aa70ddb4161 65b6abe8f3157e (779 321 B)
Feedback and results on other games are welcome, especially where it loses. And 200MB of samples ;)
|