Which Anvil engine?
Scimitar?
Anvil?
AnvilNext?
AnvilNext 2?

Pretty much all of them.

Here are samples from Origins, which is AnvilNext2. The only Anvil files i have atm on hard drive. Have fun

https://www96.zippyshare.com/v/klkEQZ94/file.html

Isn't there a file with the same name but with ".fat" extension that goes with this?

Great stuff, time to get to work :)

i deleted the link's to aviod problem...
i assume you downloaded the samples

Divinity Original Sin 2 - Samples:
GDrive folder

I did run a test, but forgot to report results but it seems to work with xtool with it being set to zlib.

FG repack has this to report:

Here's a test for a possible addition to xtool operation option

Try testing the xpreproc method on the entire game, edit the batch file to specify game location.

The game must have zlib streams, best examples would be Forza Horizon 3, GTA and so forth.

Report something like this
Only do this test below after the first one if possible, it's just up to you.

You can apply xprecomp if you want to perform further tests then srep afterwards
basically something like this:

xprecomp+srep
vs
xpreproc+xprecomp+srep

I'm interested srep's decompression memory so post the usage of memory for when xpreproc was applied and when it wasn't

:confused:

The idea here guys is for xpreproc to give xprecomp a smaller input which speeds up the process while giving roughly the same output as when it wasn't used.

-mxprecomp:
-xpreproc+xprecomp:
if you compress 4.94 GB (5,314,557,896 bytes) output, also the 1.34 GB (1,441,327,942 bytes) output, you should get the same output, the plan here is to reduce srep's memory usage and give a bit of speed and lessen IO whenever srep has created its VM file.

The Crew 2 - Open Beta
xtool not working :(

files https://www64.zippyshare.com/v/dbhy7vY9/file.html

The header file (fat) is different from the predecessors however, since this isn't lzo nor lz4, I can detect the streams directly from the dat files meaning no need to add files in xtool.ini

Try testing this program I made, it only detects streams currently, I tried to make a quick update for xtool to add support for this but I'm under a lot of work right now.

Perhaps to help, you could try figuring out what method of oodle was used from the extracted streams because I really don't have time currently.

scenaric.dat
I only tested this file

usa_world.dat
Overall size: 304237021 >> 824626021

works! :)

I figured out what they used with the spare break I had, they used the Hydra codec if I'm not mistaken. I didn't add Hydra support in xtool but I will in future.

Could you rerun test, I want to see if correct and improved information is found. I played around with the scanner a bit.

@everyone
Participate if possible, try testing this with several games that use oodle, leave out Project CARS 2 out for now since that game is encrypted.

usa_world.dat
Overall size: 314783065 >> 844382444

I had time, and i added support for this files.
Also I added parsing FAT from watch_dogs, but without (de)compression yet.
It takes time to deal with the xcompress-library.

@Razor: is xtool going to b updated.

Hi guys, long time :)

I haven't coded in a while so you can imagine:
Attachment 26918

I came up with an idea before I went hiatus which was deduplication of streams in xtool. Basically xtool permutes or disregards then removes repeated streams to improve performance. It's the same stuff that srep does only, xtool does it before srep so it improves IO and lessens memory usage.

What I want you guys to do is check out if this thing improves ratio if any and if decompression memory of srep goes down significantly, example. For examples like GTAV and lastly if there are no issues, I did rework the source a bit to make it stable.

My test input is No Man's Sky with the Beyond update (seems like devs have done a good job making sure they don't repeat game resources, wasn't the results I was not expecting...)

What do these results mean? and why is xtool 2020 producing less output compared to 2019 but somehow after using srep, the output is better?

Well as usual, I always have ideas to try to improve the xtool project for even more speed. I reviewed the project from all possible angles and found that optimisations of the program are nearing its limits meaning I cannot make the program any faster.

However then came an idea to steal some of the work srep does and make xtool involved, this resulted in the idea of making xtool detect and permute repeated streams, as a result. 2020 xtool most of the time will produce less output but when you compare it to 2019's version after using srep, the output is roughly the same or better.

You might ask, what is the point of taking some of srep's work and making it for xtool. Well this is because games over the years get endless updates, example is No Man's Sky and GTAV so there is a high chance that there are several repeated streams and as a result, if xtool doesn't take the job, it will be a job for xtool to deal with when installing a game because srep will require more decompression memory and xtool will process more streams resulting in extended installation times.

Note: I was expecting srep to produce the same result from both 2019 and 2020, because the main idea here is to improve installation times.

TL;DR

Xtool 2020 will produce less output compared to xtool 2019 because it takes some of the work of srep to reduce memory usage when installing a game while also improving speed significantly due to less streams being processed and less IO. (Theoretically at least)

Overall test results (compression):
8.95 GB (original) > 20.5 GB (xtool 2019 / 9 mins 19 secs) > 13.3 GB (srep m3f / 699mb dec memory) > 7.05 GB (lzturbo 32 b128)
8.95 GB (original) > 17.6 GB (xtool 2020 / 8 mins 4 secs) > 13.2 GB (srep m3f / 436mb dec memory) > 7.04 GB (lzturbo 32 b128)

I used "lzturbo 32" to speed up the final compression results (I'm a bit busy), but you will get the idea when you run tests yourselves.

Decompression mode is unavailable, I still have to write the code for it.

If you don't have GTAV or any other game which has live updates or a game with a lot of repetitions, you can just make a copy of the same input then give it to xtool to see what happens.

These optimisations are making me feel like Intel right now... 14nm+++++++++++++++++++++++ :o

@Razor12911
Thanks for the great job.
After completing this if you have time (also if possible), do a functional calback for your cls-diskspan library.
There are also some bugs with this library, using compressions formed by 3 methods.
Thank you!

so i guess only game with zlib for now maybe game like mad max will see improvement last time i compressed the game it was with xtool+srep:m2f+lolz memory usage was way less than srep:m3f but the final game size was 300-400mb more but game install was much more faster...
if i remember correctly m3f dec mem was around 17gb but with m2f around 3-4gb...

To be... honest... desiring a Xtool version... which will work like Pzlib v3 4ever


Thanks razor for all the works

Noted

Ok I decided to get Need For Speed The Run because I remembered that the game required a lot of decompression memory for srep to best demonstrate what I want to do with this deduplication idea.

I was surprised by the results

Xtool 2019:
Xtool 2020:

Overall test results (compression):
13.5 GB (original) > 27.1 GB (xtool 2019 / 13 mins 27 secs) > 7.51 GB (srep m3f / 5660mb dec memory) > 4.02 GB (lzturbo 32 b128)
13.5 GB (original) > 11.2 GB (xtool 2020 / 6 mins 9 secs) > 7.45 GB (srep m3f / 759mb dec memory) > 4.00 GB (lzturbo 32 b128)

Not only is xtool 2020 faster but it produced a slightly better result while reducing srep's decompression memory. That 13.5 GB > 11.2 GB is not a mistake, xtool did precompress the data but there were too many repeated streams in the input and it removed a lot of them which resulted in negative ratio.

Notes: Xtool does incur additional decoding memory but it will definitely not be 5.6GB, it will far less. Maybe 2GB but if I apply proper memory management, I can reduce it to 1GB saving an additional 3GB of ram for lolz or something else.

Good work ;)

@Razor12911
Good work! Great results!

I don't understand practically anything about compression and flow methods, etc., but CIU has XTool (old Xtool, not 2019) and ZTool, and as far as I know some of the flows (I think that's it) are not supported by Xtool.
I would like to know if in this version when it is finished it will support all flows that old ZTool and Xtool support, or it will always be necessary to keep XTool + ZTool in CIU if I use XTool 2020.
It would be great to be able to remove some of the compressors from CIU.
Thank you!

Thanks for returning to the project, deduplication is a very useful feature.
I have an idea which will reduce the required RAM for dedup. You may store some rare/large duplicated streams in a temp file, while storing small/frequent dupes in RAM - this way the excessive HDD load won't happen, cause reads will be rare and the RAM won't be used that much. 1-2 GB is a pretty big amount even for machines with 8 GB. And for users with 4 GB installation will be almost impossible, considering srep and lolz/lzma. Even with page file. So reduction/control over used RAM is a must, I think.

I'd recommend you Halo Reach for testing dedup, it has tons of duplicate streams of a different size.

maybe he can introduce the second phase in compression, then he can store forward reference count for a stream that should ideally decrease requirements memory size beyond window size and that should increase compression too.

HDDs are very slow, I recently upgraded to nvme based storage and I can feel the speed.

Dedup
 

there is also a sorting match feature that can reduce ram needed and that is as follows.

srep does a very good job finding matches that are located far away!
that in order to happen makes a dictionary!
IF you sort the files you feed srep you can actually reduce ram needed and its speed

Sorting preprocession can speedup the process and cost less ram !! and remove IO overhead because NO temps

Believe me I have several ideas of how to reduce memory usage before even relying on virtual memory. Optimisation is my middle name.

does it support stdio?

if you add stdio, it would be great..

xcompress if pure windows 10 compression, and i think it supports it by default.

Can you guys test this. I added preflate as an alternative of reflate in case if you are having issues with crc errors when you use reflate or prolonged precompression times such as this:
I'm no expert when it comes to C++ and I did my best to compile a library for xtool to use so it may have bugs.

I ran a benchmark and here are the results

benchmark results on "game1.resources" from "Dishonored Death of the Outsiders"
xtool_2019:
xtool_2020 (WIP):
PrecompX:

Dishonored Death of the Outsiders

game1.resources

When unpack function?)

Soon, maybe this week. Still busy writing the main code while looking for more ways to speed up processing :)

what is ztool and xtool and lolz?

That information is already available on these forums, search and you will find, you can not expect for others to always hold your hand and spoon feed you all the time. ;)