Game File Scanner

[elit]

I am having problem with latest R8 version if I scan games directory. Scan can get stuck on some file where progress bar reach the end but never move to another file. CPU core remain loaded but I think it encountered loop. It remain so forever(read hours). Only force quit app process is then solution. Chunk size or threads doesnt seem to help but different individual settings *may* make the app get stuck at different progress. This happen in almost every game that is big(20+gb)(GR Wildlands, Arma 3 LoW as few examples).

However if I scan that particular file where it got stuck it will pass fine. I scan Zlib, LZO and LZ4 only(for ztool). Zlib is probably ok as it alone passed game that got stuck before with +lzo +lz4 selected, those 2 can get stuck even if selected alone. I think one solution is to "tar" the directory and pass it as a single file, will try this later.

If anyone know something regarding this I would greatly appreciate it.

Btw is there an app to scan LZO like ds4zlib and ds4lz4? Something like "ds4lzo"? Those 2 are great and also show coverage percentage and most important have *zero problems*, if there was one for lzo as well I would use it instead.

PS!***THANK YOU RAZOR***!)

[KaktoR]

i have the same.

No solution for it until now.

[elit]

I tried tarring folder into single file, did not help still get stuck. There was enough free memory so its not memory leaks either. I dont know why it get stuck when same "problematic" individual file pass fine. I dont think it have to do with file itself I think app get into loop stuck.

That said zlib scan works great so its only lzo(and maybe lz4 and/or others), in fact zlib scan is better than ds4zlib including its headerless and force detection option. That one only found few mb's whereas GFS found over gb in latest arma 3.

EDIT: OK maybe I found something. In GR W game LZO kept hanging until I increased chunk size sufficiently. Anything below 256mb hanged but at 256 it consistently passed. Also I recommend 1 thread because even though it did not caused hanging, it still screwed calculation at one point - badly(at least for that game). So for LZ4 & LZO I recommend use 1 thread and start with big enough chunk size, then scale down. I will try it more in later days. Hanging looks like giving you a hint to use bigger chunk size for recompressor.

Zlib is fine at any setting and never hang but I suspect it *maybe* give false positives sometimes. Either that or recompressors are not as advanced yet. I tested on one .forge file from GR W game, scanner showed possibility of unpack by few hundred mb's(with dynamic streams), but I couldnt get it with anything I tried, at any settings. I tried precomp(+intense), ztool(pzlib:ultra, :m3, :x etc...), pzlib3 but no go I got exact same file size.

EDIT2: Also DS4LZ4 give different size for Mirrors Edge Catalyst. GFS find about 7gb(packed) and DS4LZ4 almost 11gb(packed)(and hang at 95%). Both same settings. I dont know which one to believe now.

[Razor12911]

Originally Posted by elit

I am having problem with latest R8 version if I scan games directory. Scan can get stuck on some file where progress bar reach the end but never move to another file. CPU core remain loaded but I think it encountered loop. It remain so forever(read hours). Only force quit app process is then solution. Chunk size or threads doesnt seem to help but different individual settings *may* make the app get stuck at different progress. This happen in almost every game that is big(20+gb)(GR Wildlands, Arma 3 LoW as few examples).

However if I scan that particular file where it got stuck it will pass fine. I scan Zlib, LZO and LZ4 only(for ztool). Zlib is probably ok as it alone passed game that got stuck before with +lzo +lz4 selected, those 2 can get stuck even if selected alone. I think one solution is to "tar" the directory and pass it as a single file, will try this later.

If anyone know something regarding this I would greatly appreciate it.

Btw is there an app to scan LZO like ds4zlib and ds4lz4? Something like "ds4lzo"? Those 2 are great and also show coverage percentage and most important have *zero problems*, if there was one for lzo as well I would use it instead.

PS!***THANK YOU RAZOR***!)

Originally Posted by KaktoR

i have the same.

No solution for it until now.

Originally Posted by elit

I tried tarring folder into single file, did not help still get stuck. There was enough free memory so its not memory leaks either. I dont know why it get stuck when same "problematic" individual file pass fine. I dont think it have to do with file itself I think app get into loop stuck.

That said zlib scan works great so its only lzo(and maybe lz4 and/or others), in fact zlib scan is better than ds4zlib including its headerless and force detection option. That one only found few mb's whereas GFS found over gb in latest arma 3.

EDIT: OK maybe I found something. In GR W game LZO kept hanging until I increased chunk size sufficiently. Anything below 256mb hanged but at 256 it consistently passed. Also I recommend 1 thread because even though it did not caused hanging, it still screwed calculation at one point - badly(at least for that game). So for LZ4 & LZO I recommend use 1 thread and start with big enough chunk size, then scale down. I will try it more in later days. Hanging looks like giving you a hint to use bigger chunk size for recompressor.

Zlib is fine at any setting and never hang but I suspect it *maybe* give false positives sometimes. Either that or recompressors are not as advanced yet. I tested on one .forge file from GR W game, scanner showed possibility of unpack by few hundred mb's(with dynamic streams), but I couldnt get it with anything I tried, at any settings. I tried precomp(+intense), ztool(pzlib:ultra, :m3, :x etc...), pzlib3 but no go I got exact same file size.

EDIT2: Also DS4LZ4 give different size for Mirrors Edge Catalyst. GFS find about 7gb(packed) and DS4LZ4 almost 11gb(packed)(and hang at 95%). Both same settings. I dont know which one to believe now.

No longer updating my programs, sorry. DS4LZ4 scans are false, almost all the time.

[elit]

Originally Posted by Razor12911

No longer updating my programs, sorry. DS4LZ4 scans are false, almost all the time.

Thanks for the info. Yes I read about it and understand you position. Really sorry about what happened.

If I may ask you one last thing, I am using GFS together in conjunction with ztool. These 2 tools are just as cool and as important for repacks as srep/freearc by this Bulat Zigurat guy(or whatever was his name, but his tools are truly fantastic as well).

So I scan the game folder and then use ztool+srep/freearc. But I noticed that GFS have "dynamic streams" option which can find significantly more data. But is ztool also able to find and process dynamic streams? Because if not there is no point scanning for them and can save time. And if yes do I need specific option?(So far I found that I get almost always same size regardless of whether I use :pzlib:ultra or no option.)

Thank you again for these cool tools & best regards.

[Joe Forster/STA]

(If you write ":p" in your post, it will be displayed as a smiley like this: . For plain text, enclose it into "noparse" tags instead, like this: "[noparse]:p[/noparse]".)

[elit]

Ok so gave it some thoughts, and realized... "why not just use ztool for scan directly"?
Because, to me it seems GFS and ztool are about same speed and only difference is disk writing process on the ztool side. And ztool not only never hang, it give you 100% accurate result of what you would actually get during recompression!

So, this calls for cmd scripting, I only tried basic one if it work on single file. And it does:
calc.png

Just download dd for windows and pipe it to ztool. Make sure you /dev/null output.
Now, to make it more complex batch script for whole directory.... I guess tarring+pipelining would do(or 7z -m0 "store" + pipe), but for now you get an idea. It doesnt give you all the info, but then again what more you need than seeing size differences + this is 100% accurate to real result - no false positives and no missed streams.

[elit]

Alright so I made the script. Edit it for your game path, change ztool parameters as needed and run it. It need DD command for windows which you can get here:
http://www.chrysocome.net/dd.
Next you need 7zip, specifically 7z.exe and 7z.dll only, and ztool.

Place everything in one folder, including my attached script scan.zip and you are good to go. This script will show time at the beginning and the end of work(not shown on the pic as I added it later) and it will show original as well as pre-processed size at the end:
scrpt.png

This script is slower than GFS as I found, probably because inefficient serial piping, but still "ok" and reliable. Unlike GFS, feel free to set up as many threads on ztool as you can to speed it up without reliability cost.

I recommend this for plz4 and plzo since they hang, you can keep using pzlib on GFS(with dynamic streams on) to get more or less same accuracy while being quicker. But if you want to have exact size that you would get using ztool, you can still use this for pzlib as well.

PS: since this doesnt utilize disk fully(like GFS do), you could have say 3 different batch script versions in the folder, one for pzlib, plz4 and plzo respectively and run all 3 of them at once.

[Razor12911]

Nice, problem is GFS finds more stuff, stuff that was incomplete in ztool. the leaked version for example cannot process crilayla streams, nor lzo streams from unreal engine games and so forth so yes, your alternative method works but to a certain extent

[elit]

Originally Posted by Razor12911

Nice, problem is GFS finds more stuff, stuff that was incomplete in ztool. the leaked version for example cannot process crilayla streams, nor lzo streams from unreal engine games and so forth so yes, your alternative method works but to a certain extent

Indeed it look so. With that said though, to us regular people, the only relevant streams are the ones we can actually process. Even if GFS can find more, it is of no use if we lack the tool to process all of them. Thats why I think using ztool is(for us) better - to show us exactly what we get during actual packing.

[nik1967]

Why I did not show the size? And memory size for scanning?

[KaktoR]

Same here.

For some it shows, for some not

[felice2011]

Originally Posted by nik1967

Why I did not show the size? And memory size for scanning?

I always think about the same problem, the only solution.



Enter the decimal point...change "," to "." and try.

[nik1967]

felice2011, thank you very much! Helped.

[KaktoR]

Thank you, that helped me too