usa_world.dat
Overall size: 304237021 >> 824626021

works! :)

I figured out what they used with the spare break I had, they used the Hydra codec if I'm not mistaken. I didn't add Hydra support in xtool but I will in future.

Could you rerun test, I want to see if correct and improved information is found. I played around with the scanner a bit.

@everyone
Participate if possible, try testing this with several games that use oodle, leave out Project CARS 2 out for now since that game is encrypted.

usa_world.dat
Overall size: 314783065 >> 844382444

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

@Razor: is xtool going to b updated.

Hi guys, long time :)

I haven't coded in a while so you can imagine:
Attachment 26918

I came up with an idea before I went hiatus which was deduplication of streams in xtool. Basically xtool permutes or disregards then removes repeated streams to improve performance. It's the same stuff that srep does only, xtool does it before srep so it improves IO and lessens memory usage.

What I want you guys to do is check out if this thing improves ratio if any and if decompression memory of srep goes down significantly, example. For examples like GTAV and lastly if there are no issues, I did rework the source a bit to make it stable.

My test input is No Man's Sky with the Beyond update (seems like devs have done a good job making sure they don't repeat game resources, wasn't the results I was not expecting...)

What do these results mean? and why is xtool 2020 producing less output compared to 2019 but somehow after using srep, the output is better?

Well as usual, I always have ideas to try to improve the xtool project for even more speed. I reviewed the project from all possible angles and found that optimisations of the program are nearing its limits meaning I cannot make the program any faster.

However then came an idea to steal some of the work srep does and make xtool involved, this resulted in the idea of making xtool detect and permute repeated streams, as a result. 2020 xtool most of the time will produce less output but when you compare it to 2019's version after using srep, the output is roughly the same or better.

You might ask, what is the point of taking some of srep's work and making it for xtool. Well this is because games over the years get endless updates, example is No Man's Sky and GTAV so there is a high chance that there are several repeated streams and as a result, if xtool doesn't take the job, it will be a job for xtool to deal with when installing a game because srep will require more decompression memory and xtool will process more streams resulting in extended installation times.

Note: I was expecting srep to produce the same result from both 2019 and 2020, because the main idea here is to improve installation times.

TL;DR

Xtool 2020 will produce less output compared to xtool 2019 because it takes some of the work of srep to reduce memory usage when installing a game while also improving speed significantly due to less streams being processed and less IO. (Theoretically at least)

Overall test results (compression):
8.95 GB (original) > 20.5 GB (xtool 2019 / 9 mins 19 secs) > 13.3 GB (srep m3f / 699mb dec memory) > 7.05 GB (lzturbo 32 b128)
8.95 GB (original) > 17.6 GB (xtool 2020 / 8 mins 4 secs) > 13.2 GB (srep m3f / 436mb dec memory) > 7.04 GB (lzturbo 32 b128)

I used "lzturbo 32" to speed up the final compression results (I'm a bit busy), but you will get the idea when you run tests yourselves.

Decompression mode is unavailable, I still have to write the code for it.

If you don't have GTAV or any other game which has live updates or a game with a lot of repetitions, you can just make a copy of the same input then give it to xtool to see what happens.

These optimisations are making me feel like Intel right now... 14nm+++++++++++++++++++++++ :o

@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!

so i guess only game with zlib for now maybe game like mad max will see improvement last time i compressed the game it was with xtool+srep:m2f+lolz memory usage was way less than srep:m3f but the final game size was 300-400mb more but game install was much more faster...
if i remember correctly m3f dec mem was around 17gb but with m2f around 3-4gb...

To be... honest... desiring a Xtool version... which will work like Pzlib v3 4ever


Thanks razor for all the works

Noted

Ok I decided to get Need For Speed The Run because I remembered that the game required a lot of decompression memory for srep to best demonstrate what I want to do with this deduplication idea.

I was surprised by the results

Xtool 2019:
Xtool 2020:

Overall test results (compression):
13.5 GB (original) > 27.1 GB (xtool 2019 / 13 mins 27 secs) > 7.51 GB (srep m3f / 5660mb dec memory) > 4.02 GB (lzturbo 32 b128)
13.5 GB (original) > 11.2 GB (xtool 2020 / 6 mins 9 secs) > 7.45 GB (srep m3f / 759mb dec memory) > 4.00 GB (lzturbo 32 b128)

Not only is xtool 2020 faster but it produced a slightly better result while reducing srep's decompression memory. That 13.5 GB > 11.2 GB is not a mistake, xtool did precompress the data but there were too many repeated streams in the input and it removed a lot of them which resulted in negative ratio.

Notes: Xtool does incur additional decoding memory but it will definitely not be 5.6GB, it will far less. Maybe 2GB but if I apply proper memory management, I can reduce it to 1GB saving an additional 3GB of ram for lolz or something else.

Good work ;)

@Razor12911
Good work! Great results!

I don't understand practically anything about compression and flow methods, etc., but CIU has XTool (old Xtool, not 2019) and ZTool, and as far as I know some of the flows (I think that's it) are not supported by Xtool.
I would like to know if in this version when it is finished it will support all flows that old ZTool and Xtool support, or it will always be necessary to keep XTool + ZTool in CIU if I use XTool 2020.
It would be great to be able to remove some of the compressors from CIU.
Thank you!

Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.