TARZ — Texture Adaptive RANS

[panker1992]

Hello everyone,

During the past 5 months i have been working on several stuff.

Including Heavy deduplications engines, compression algorithms, precompression and other adjustments my archiver has already build in.

Today i am glad to present to you a new compression tool called, Tarz.


Tarz, is a lolz replacement which i created several weeks ago.
During my recent work on OGG/Opus data i found a way to see how modeling on data really works.
And therefore i decided to create something that can be used to compress Texture heavy Data.

I hope you enjoy using this tool.

In order for you to use this tool, ram is required because it has a build-in LRM.
Which fetches the data from a hefty distance away all while a pricing model creator is running for it.

Take notice that each Thread you pop, it will have its own memory, its own set of models and so on. so using multihtreading will heavily creeple the ratio on this tool.

[Masquerade]

Thanks
Been excited for this one...

[kj911]

Originally Posted by panker1992

Take notice that each Thread you pop, it will have its own memory, its own set of models and so on. so using multihtreading will heavily creeple the ratio on this tool.

Am I reading this correctly? If I use a single thread for compression, do I end up with a worse final compressed size compared to classic LZ algorithms—because there is no single-pass solid compression—meaning that running it on multiple threads would actually yield a better compression ratio? In the case of LOLZ/LZMA, does using a single thread produce the best results, while using multiple threads makes it faster but results in poorer performance (in terms of compression ratio)?

Are there any comparative test results regarding compression ratios for LOLZ/LZMA and Razor vs. Tarz?

[dixen]

Interesting project, thank you. It compresses very close to LOLZ, albeit more slowly. I would like to try it with settings equivalent to mtt1 tt2 mt8 mc 32. How can they be set?

[panker1992]

Unfortunately the other day i was so excited to post this, that i forgot to remeasure some variants.

i am working on making it a bit more adaptive than its already is.

Let me explain how this works, and let me explain how i think lolz works.

Tarz has 2 main operations, and it all comes down as to hwo you want to use it.

using, 1 single threads you actually force a single case of models which will span from the beginning of the file until the end of it, and that model will have all known cases for all the files, textures and records it can find and predict.

When using more than one thread this will go away, it split the data in as many equal chunks as you put threads, and in return it will actually make a number of predictions in those ministreams hence you need as many cores to decompress as to compress.

i am actually working on adaptation to give you an already close to lolz ratio as possible.


BTW Tarz isn't a lolz clone, not a copy paste , nor a Reverse Engineering Project.

It is written from scratch in Delphi, using some Advanced techniques i learnt from modelling data.

To-Do:

Advanced AutoParser - Close Gap to lolz
Make it reach further to provide better coverage, lolz can see further it has LDMF-LRM techniques inside which i lack at the moment.

I cannot guarantee the next generation of Tarz will be backwards compatible since this is a demo, Version 2 will be the stable release in my mind

[kj911]

I ran a quick test using the standard 100MB test archive (a file compressed with the `XZLib+srep:m3f:a0` chain).

If you want to keep things very simple—and don't mind a 10–20% drop in decompression speed—you can use: `d128:l9:cm1`.

Using `cm0` results at the cost of a roughly 1% loss in compression ratio.

LOLZ is 4–5 times faster than TARZ, even when running on a single thread. It is faster with the aforementioned settings; however, I haven't tested it yet on files larger than 1GB, due to considerations regarding thread utilization and memory requirements.

You can see the results of several experiments in the text file below. The patched `cls-filter` also works on XP! It is slow in that environment, though; its benefits are only truly noticeable on a machine with at least 2–4 CPU cores/threads—even if you happen to be running 32-bit XP or Windows 7.

[panker1992]

Unfortunately, i failed to properly introduce this tool and its capabilities because i was too excited to make it available.

I have streamlined the vast available options, to make some completely static, and others are now properly utilized.

Mins and Maxes are properly displayed while a help is guiding you how to use this properly.


defaults can help you reach lolz ratio, also there are some vars that while increased also increase waiting time exponentially, take care to find what your best favorite is.

increase d or t and then start to play with other variables

[FitGirl]

Thank you, finally something to play with

[panker1992]

Originally Posted by FitGirl

Thank you, finally something to play with

While MT breaks ratio a little bit, on the other hand it increases decompression speed.

I am planning on adding a padded decompression liek the one lolz has which is block based.

mine is frame based.

[FitGirl]

I'd also check other models lolz and alike compressors use. Modern most popular engines like UE or Unity have all data mixes, so textures are go after char models, audio, etc. So if tarz models transform the data the bad way this may decrease final compression ratio.

[panker1992]

Let me show you how nasty my tool really is.

Check this out. this is Memoriapolis packed with my stuff and your repack



What you say above is true, which means you need to know what you are dealing with. Tarz doesn;t know per se and neither lolz.

There is also the switch to kill the models

[FitGirl]

Good ratio! And what about decomp speed?

btw, why there are so many "lolz" strings inside the exe?

Edit: I've chosen the wrong precomp params for Memoriapolis, will do a proper test later.

Still, TARZ is a very promising project even if it relies on LOLZ (which is based on ROLZ in it's turn), so even if REA was used, it's good - considering that some other older compressors like NanoZIP are now being recovered the same way.
So, godspeed!

[panker1992]

i have never used reverse engineering techniques on LOLZ and even then Tarz is written in Delphi, from what i think lolz must be either C or CPP.

However i have used reverse logic on the tool. i basically recreated the tool by individually compressing smaller inputs and comparing them to their original counterparts.


This along with modelling logic i have from several other projects allowed for the creation of Tarz, which uses RANS as backend and not ROLZ.

All the ratio comes from Models, and their prediction, i have also developed other tools, foxlz and veloz. Native to my archiver and maybe i will bring them to FA.

Decode Speeds as you have asked. Here a picture.

[FitGirl]

I know about your other projects, but yeah, that was a surprise to see strings like this (LolzDecodePipe, LolzThreadDecoder, etc) - guess you used them as internal names as it was based on lolz



I also tested TARZ on that Memoriaplis game, but properly decompressed this time.

1705382191 mem-tarz.arc
1664803071 mem-lolz.arc

Alas, but when chained with xtool+srep tarz could not unpack the arc (it worked fine on decomp on smaller/simple samples). Tried both cls and tarz.exe d options.

Testing Memoriapolis_Data\globalgamemanagers (796500 bytes)
Testing Memoriapolis_Data\level0 (2696 bytes)
Testing Memoriapolis_Data\level1 (3552 bytes)
Testing Memoriapolis_Data\level10 (255960680 bytes)

ERROR! Decompression problem: unexpected end of file - or I/O error

How to address this?

Also, do you plan to expand threads to N/unlimited from current 16? That was the limit of lolz, ProFrager told then that there are not much non-server CPU which have more threads, but now 16t is already very popular with more threaded CPUs arriving every year.

I also noticed that TARZ is more stable when it comes to compression speed. Lolz can severely slow down close to the end of compression, while tarz stays relatively the same. But need more larger samples to confirm.

[kj911]

As far as I know, LOLZ uses RZM filters—evidently enhanced with additional filters and concepts—which are then combined with LZNA compression, incorporating some form of CM (Context Mixing). This is why the codec is slow during compression but fast during decompression.

I haven't really encountered a scenario where it slows down specifically toward the end of the compression process; however, LDMF could be a potential cause—which is to be expected if it has to search for, and subsequently finds, a large number of matches. I have, however, experienced extreme slowdowns during certain phases, likely due to the CM component. The game *SCPT* exhibits this behavior with some of its data files (specifically, a certain section of one of the audio archives).

Regarding multithreading, is it possible to make the processes independent? In other words, if I compress using only 1 or 2–4 threads, could it decompress the data using, say, 16 threads (or vice versa)? Also, is there a way to control memory usage during decompression—similar to how it’s handled in LOLZ—accepting a slight loss in speed in exchange for more stable operation?