TARZ — Texture Adaptive RANS

[panker1992]

Hello everyone,

During the past 5 months i have been working on several stuff.

Including Heavy deduplications engines, compression algorithms, precompression and other adjustments my archiver has already build in.

Today i am glad to present to you a new compression tool called, Tarz.


Tarz, is a lolz replacement which i created several weeks ago.
During my recent work on OGG/Opus data i found a way to see how modeling on data really works.
And therefore i decided to create something that can be used to compress Texture heavy Data.

I hope you enjoy using this tool.

In order for you to use this tool, ram is required because it has a build-in LRM.
Which fetches the data from a hefty distance away all while a pricing model creator is running for it.

Take notice that each Thread you pop, it will have its own memory, its own set of models and so on. so using multihtreading will heavily creeple the ratio on this tool.

[Masquerade]

Thanks
Been excited for this one...

[kj911]

Originally Posted by panker1992

Take notice that each Thread you pop, it will have its own memory, its own set of models and so on. so using multihtreading will heavily creeple the ratio on this tool.

Am I reading this correctly? If I use a single thread for compression, do I end up with a worse final compressed size compared to classic LZ algorithms—because there is no single-pass solid compression—meaning that running it on multiple threads would actually yield a better compression ratio? In the case of LOLZ/LZMA, does using a single thread produce the best results, while using multiple threads makes it faster but results in poorer performance (in terms of compression ratio)?

Are there any comparative test results regarding compression ratios for LOLZ/LZMA and Razor vs. Tarz?

[dixen]

Interesting project, thank you. It compresses very close to LOLZ, albeit more slowly. I would like to try it with settings equivalent to mtt1 tt2 mt8 mc 32. How can they be set?

[panker1992]

Unfortunately the other day i was so excited to post this, that i forgot to remeasure some variants.

i am working on making it a bit more adaptive than its already is.

Let me explain how this works, and let me explain how i think lolz works.

Tarz has 2 main operations, and it all comes down as to hwo you want to use it.

using, 1 single threads you actually force a single case of models which will span from the beginning of the file until the end of it, and that model will have all known cases for all the files, textures and records it can find and predict.

When using more than one thread this will go away, it split the data in as many equal chunks as you put threads, and in return it will actually make a number of predictions in those ministreams hence you need as many cores to decompress as to compress.

i am actually working on adaptation to give you an already close to lolz ratio as possible.


BTW Tarz isn't a lolz clone, not a copy paste , nor a Reverse Engineering Project.

It is written from scratch in Delphi, using some Advanced techniques i learnt from modelling data.

To-Do:

Advanced AutoParser - Close Gap to lolz
Make it reach further to provide better coverage, lolz can see further it has LDMF-LRM techniques inside which i lack at the moment.

I cannot guarantee the next generation of Tarz will be backwards compatible since this is a demo, Version 2 will be the stable release in my mind

[kj911]

I ran a quick test using the standard 100MB test archive (a file compressed with the `XZLib+srep:m3f:a0` chain).

If you want to keep things very simple—and don't mind a 10–20% drop in decompression speed—you can use: `d128:l9:cm1`.

Using `cm0` results at the cost of a roughly 1% loss in compression ratio.

LOLZ is 4–5 times faster than TARZ, even when running on a single thread. It is faster with the aforementioned settings; however, I haven't tested it yet on files larger than 1GB, due to considerations regarding thread utilization and memory requirements.

You can see the results of several experiments in the text file below. The patched `cls-filter` also works on XP! It is slow in that environment, though; its benefits are only truly noticeable on a machine with at least 2–4 CPU cores/threads—even if you happen to be running 32-bit XP or Windows 7.

[panker1992]

Unfortunately, i failed to properly introduce this tool and its capabilities because i was too excited to make it available.

I have streamlined the vast available options, to make some completely static, and others are now properly utilized.

Mins and Maxes are properly displayed while a help is guiding you how to use this properly.


defaults can help you reach lolz ratio, also there are some vars that while increased also increase waiting time exponentially, take care to find what your best favorite is.

increase d or t and then start to play with other variables