Hi guys, long time
I haven't coded in a while so you can imagine:
96806729_3811479972226955_3580484423860092928_n.jpg
I came up with an idea before I went hiatus which was deduplication of streams in xtool. Basically xtool permutes or disregards then removes repeated streams to improve performance. It's the same stuff that srep does only, xtool does it before srep so it improves IO and lessens memory usage.
What I want you guys to do is check out if this thing improves ratio if any and if decompression memory of srep goes down significantly, example. For examples like GTAV and lastly if there are no issues, I did rework the source a bit to make it stable.
My test input is No Man's Sky with the Beyond update (seems like devs have done a good job making sure they don't repeat game resources, wasn't the results I was not expecting...)
Code:
xtool 2019:
Compressed 104 files, 9,610,947,737 => 22,079,473,953 bytes. Ratio 229.73%
Compression time: cpu 12.20 sec/real 559.10 sec = 2%. Speed 17.19 mB/s
srep393b:
100%: 22,079,475,924 -> 14,306,151,312: 64.79%. Cpu 123 mb/s (171.078 sec), real 126 mb/s (167.671 sec) = 102%. Remains 00:00
Decompression memory is 699 mb. 6,413,440 matches = 102,615,040 bytes = 0.72% of file
lzturbo:
Compressed 1 file, 14,306,163,456 => 7,573,816,911 bytes. Ratio 52.94%
Compression time: cpu 16.19 sec/real 255.92 sec = 6%. Speed 55.90 mB/s
Code:
xtool 2020:
Compressed 104 files, 9,610,947,737 => 18,987,442,809 bytes. Ratio 197.56%
Compression time: cpu 10.94 sec/real 483.97 sec = 2%. Speed 19.86 mB/s
srep393b:
100%: 18,987,444,781 -> 14,276,551,933: 75.19%. Cpu 113 mb/s (160.000 sec), real 114 mb/s (158.234 sec) = 101%. Remains 00:00
Decompression memory is 436 mb. 4,696,348 matches = 75,141,568 bytes = 0.53% of file
lzturbo:
Compressed 1 file, 14,276,560,829 => 7,565,632,525 bytes. Ratio 52.99%
Compression time: cpu 16.86 sec/real 250.04 sec = 7%. Speed 57.10 mB/s
What do these results mean? and why is xtool 2020 producing less output compared to 2019 but somehow after using srep, the output is better?
Well as usual, I always have ideas to try to improve the xtool project for even more speed. I reviewed the project from all possible angles and found that optimisations of the program are nearing its limits meaning I cannot make the program any faster.
However then came an idea to steal some of the work srep does and make xtool involved, this resulted in the idea of making xtool detect and permute repeated streams, as a result. 2020 xtool most of the time will produce less output but when you compare it to 2019's version after using srep, the output is roughly the same or better.
You might ask, what is the point of taking some of srep's work and making it for xtool. Well this is because games over the years get endless updates, example is No Man's Sky and GTAV so there is a high chance that there are several repeated streams and as a result, if xtool doesn't take the job, it will be a job for xtool to deal with when installing a game because srep will require more decompression memory and xtool will process more streams resulting in extended installation times.
Note: I was expecting srep to produce the same result from both 2019 and 2020, because the main idea here is to improve installation times.
TL;DR
Xtool 2020 will produce less output compared to xtool 2019 because it takes some of the work of srep to reduce memory usage when installing a game while also improving speed significantly due to less streams being processed and less IO. (Theoretically at least)
Overall test results (compression):
8.95 GB (original) > 20.5 GB (xtool 2019 / 9 mins 19 secs) > 13.3 GB (srep m3f / 699mb dec memory) > 7.05 GB (lzturbo 32 b128)
8.95 GB (original) > 17.6 GB (xtool 2020 / 8 mins 4 secs) > 13.2 GB (srep m3f / 436mb dec memory) > 7.04 GB (lzturbo 32 b128)
I used "lzturbo 32" to speed up the final compression results (I'm a bit busy), but you will get the idea when you run tests yourselves.
Decompression mode is
unavailable, I still have to write the code for it.
If you don't have GTAV or any other game which has live updates or a game with a lot of repetitions, you can just make a copy of the same input then give it to xtool to see what happens.
These optimisations are making me feel like Intel right now... 14nm+++++++++++++++++++++++ 