Game File Scanner

[Razor12911]

Another file scanner for games, helps you figure out what compression algorithm was used.

[Mini]

good work!

[Gupta]

Can you add zlib level detector i mean you already have one

[Razor12911]

Update available

Changes

-Added crilayla detector. (Might be buggy) Screenshot below shows final size of program vs the unpacked data using Criware CriPack GUI.

[unarc 125]

Works good !
Just a little bug if scan button is pressed and input is nothing then the button get disabled and never get enabled back even if we give input.
And btw lz4 scanning is same good as zlib ?

[Razor12911]

Can you add zlib level detector i mean you already have one

I made zlib scanner only focus on deflate streams (no zlib header)

Works good !
Just a little bug if scan button is pressed and input is nothing then the button get disabled and never get enabled back even if we give input.
And btw lz4 scanning is same good as zlib ?

Then you have to terminate program and try again with higher chunk size or by disabling some algorithms. Happy belated birthday btw :)

[unarc 125]

Thank you :)
And if you increase the width and decrease in height would be good

[Razor12911]

Thank you :)
And if you increase the width and decrease in height would be good

Done.

[unarc 125]

Sweet !

[Crynuvo]

Many Thanks For The Tool! :)

[Jiva newstone]

best of scanning

[KaktoR]

We can not scan a whole folder instead of a single file?

[Razor12911]

Update available

Changes

-Fixed a few bugs
-Updated Crilayla detection for more speed and accuracy
-Updated zlib and lz4 detection for less false positives
-Removed Dishorned 2 support (headache...)
-Added wav, bink and vp6 detection

[omdj]

We can not scan a whole folder instead of a single file?

Yes, it would be more practical.

[Razor12911]

Update available


Changes

-Added directory scanning option
-Readded Dishonored 2 support
-Added Naruto shippuden ninja storm support

[KaktoR]

Thanks.

Which one is correct?

[78372]

Check again with headerless and force detection turned on

[KaktoR]

Thanks for that info.

It is quiet even now (~200 less).

[Razor12911]

Update available

Changes

-Added FIFA 17 support

[EzzEldin16]

is that somekind of a bug or what ?
i was scanning NFS:Shift 2 Unleashed and i don't know what kind of streams are there
all what was showing size is <Overall results:Size: 3.63 GB >> 3.63 GB> so tell what to do
am i doing something wrong ?
is it another type of data streams or what ?

[Razor12911]

is that somekind of a bug or what ?
i was scanning NFS:Shift 2 Unleashed and i don't know what kind of streams are there
all what was showing size is <Overall results:Size: 3.63 GB >> 3.63 GB> so tell what to do
am i doing something wrong ?
is it another type of data streams or what ?

I think that game is encrypted just like Project CARS since it's developed by the same people.

[Razor12911]

Can you add zlib level detector i mean you already have one

Update available

Changes

-Added zlib level detector, it's quite a bit slow but did my best for more speed, which led to inaccuracy on some input.

[dixen]

Hi guys!
http://f2.s.qip.ru/jjb2KIJH.png
How decompress it?

[EzzEldin16]

What do you mean by decompress ?

[JustFun]

He probably thinks this is compressor, this tool is only for scanning game files its not used to compress them.

[EzzEldin16]

@JustFun probably

[dixen]

Guys, I know it's the scanner. Therefore I asked how to decompress what is selected in the screenshot

[EzzEldin16]

AGAIN what do you mean by decompress show us what you have is it an a game or an archive or what is it you can't extract data streams can you ?

[lrgame1983]

damn this is nice, thanks razor12911

[L0v3craft]

Hi Razor12911. Thanks for this program ! I have just a little request. Is possible to add a button to save a report of the scanned files ?

For example, I have 945 files with .umap extension. I know that not all of them have zlib streams. I would like to get a report(a txt file) with your program, a list of the only files that have some stream inside. So I can know which files needs to be pumped with pzlib.

[elit]

I am having problem with latest R8 version if I scan games directory. Scan can get stuck on some file where progress bar reach the end but never move to another file. CPU core remain loaded but I think it encountered loop. It remain so forever(read hours). Only force quit app process is then solution. Chunk size or threads doesnt seem to help but different individual settings *may* make the app get stuck at different progress. This happen in almost every game that is big(20+gb)(GR Wildlands, Arma 3 LoW as few examples).

However if I scan that particular file where it got stuck it will pass fine. I scan Zlib, LZO and LZ4 only(for ztool). Zlib is probably ok as it alone passed game that got stuck before with +lzo +lz4 selected, those 2 can get stuck even if selected alone. I think one solution is to "tar" the directory and pass it as a single file, will try this later.

If anyone know something regarding this I would greatly appreciate it.

Btw is there an app to scan LZO like ds4zlib and ds4lz4? Something like "ds4lzo"? Those 2 are great and also show coverage percentage and most important have *zero problems*, if there was one for lzo as well I would use it instead.

PS:(!***THANK YOU RAZOR***!)

[KaktoR]

i have the same.

No solution for it until now.

[elit]

I tried tarring folder into single file, did not help still get stuck. There was enough free memory so its not memory leaks either. I dont know why it get stuck when same "problematic" individual file pass fine. I dont think it have to do with file itself I think app get into loop stuck.

That said zlib scan works great so its only lzo(and maybe lz4 and/or others), in fact zlib scan is better than ds4zlib including its headerless and force detection option. That one only found few mb's whereas GFS found over gb in latest arma 3.

EDIT: OK maybe I found something. In GR W game LZO kept hanging until I increased chunk size sufficiently. Anything below 256mb hanged but at 256 it consistently passed. Also I recommend 1 thread because even though it did not caused hanging, it still screwed calculation at one point - badly(at least for that game). So for LZ4 & LZO I recommend use 1 thread and start with big enough chunk size, then scale down. I will try it more in later days. Hanging looks like giving you a hint to use bigger chunk size for recompressor.

Zlib is fine at any setting and never hang but I suspect it *maybe* give false positives sometimes. Either that or recompressors are not as advanced yet. I tested on one .forge file from GR W game, scanner showed possibility of unpack by few hundred mb's(with dynamic streams), but I couldnt get it with anything I tried, at any settings. I tried precomp(+intense), ztool(pzlib:ultra, :m3, :x etc...), pzlib3 but no go I got exact same file size.

EDIT2: Also DS4LZ4 give different size for Mirrors Edge Catalyst. GFS find about 7gb(packed) and DS4LZ4 almost 11gb(packed)(and hang at 95%). Both same settings. I dont know which one to believe now.

[Razor12911]

I am having problem with latest R8 version if I scan games directory. Scan can get stuck on some file where progress bar reach the end but never move to another file. CPU core remain loaded but I think it encountered loop. It remain so forever(read hours). Only force quit app process is then solution. Chunk size or threads doesnt seem to help but different individual settings *may* make the app get stuck at different progress. This happen in almost every game that is big(20+gb)(GR Wildlands, Arma 3 LoW as few examples).

However if I scan that particular file where it got stuck it will pass fine. I scan Zlib, LZO and LZ4 only(for ztool). Zlib is probably ok as it alone passed game that got stuck before with +lzo +lz4 selected, those 2 can get stuck even if selected alone. I think one solution is to "tar" the directory and pass it as a single file, will try this later.

If anyone know something regarding this I would greatly appreciate it.

Btw is there an app to scan LZO like ds4zlib and ds4lz4? Something like "ds4lzo"? Those 2 are great and also show coverage percentage and most important have *zero problems*, if there was one for lzo as well I would use it instead.

PS:(!***THANK YOU RAZOR***!)



i have the same.

No solution for it until now.



I tried tarring folder into single file, did not help still get stuck. There was enough free memory so its not memory leaks either. I dont know why it get stuck when same "problematic" individual file pass fine. I dont think it have to do with file itself I think app get into loop stuck.

That said zlib scan works great so its only lzo(and maybe lz4 and/or others), in fact zlib scan is better than ds4zlib including its headerless and force detection option. That one only found few mb's whereas GFS found over gb in latest arma 3.

EDIT: OK maybe I found something. In GR W game LZO kept hanging until I increased chunk size sufficiently. Anything below 256mb hanged but at 256 it consistently passed. Also I recommend 1 thread because even though it did not caused hanging, it still screwed calculation at one point - badly(at least for that game). So for LZ4 & LZO I recommend use 1 thread and start with big enough chunk size, then scale down. I will try it more in later days. Hanging looks like giving you a hint to use bigger chunk size for recompressor.

Zlib is fine at any setting and never hang but I suspect it *maybe* give false positives sometimes. Either that or recompressors are not as advanced yet. I tested on one .forge file from GR W game, scanner showed possibility of unpack by few hundred mb's(with dynamic streams), but I couldnt get it with anything I tried, at any settings. I tried precomp(+intense), ztool(pzlib:ultra, :m3, :x etc...), pzlib3 but no go I got exact same file size.

EDIT2: Also DS4LZ4 give different size for Mirrors Edge Catalyst. GFS find about 7gb(packed) and DS4LZ4 almost 11gb(packed)(and hang at 95%). Both same settings. I dont know which one to believe now.

No longer updating my programs, sorry. DS4LZ4 scans are false, almost all the time.

[elit]

No longer updating my programs, sorry. DS4LZ4 scans are false, almost all the time.

Thanks for the info. Yes I read about it and understand you position. Really sorry about what happened.

If I may ask you one last thing, I am using GFS together in conjunction with ztool. These 2 tools are just as cool and as important for repacks as srep/freearc by this Bulat Zigurat guy(or whatever was his name, but his tools are truly fantastic as well).

So I scan the game folder and then use ztool+srep/freearc. But I noticed that GFS have "dynamic streams" option which can find significantly more data. But is ztool also able to find and process dynamic streams? Because if not there is no point scanning for them and can save time. And if yes do I need specific option?(So far I found that I get almost always same size regardless of whether I use :pzlib:ultra or no option.)

Thank you again for these cool tools & best regards.

[Joe Forster/STA]

(If you write ":p" in your post, it will be displayed as a smiley like this: :p. For plain text, enclose it into "noparse" tags instead, like this: ":p".)

[elit]

Ok so gave it some thoughts, and realized... "why not just use ztool for scan directly"?
Because, to me it seems GFS and ztool are about same speed and only difference is disk writing process on the ztool side. And ztool not only never hang, it give you 100% accurate result of what you would actually get during recompression!

So, this calls for cmd scripting, I only tried basic one if it work on single file. And it does:
19361

Just download dd for windows and pipe it to ztool. Make sure you /dev/null output.
Now, to make it more complex batch script for whole directory.... I guess tarring+pipelining would do(or 7z -m0 "store" + pipe), but for now you get an idea. It doesnt give you all the info, but then again what more you need than seeing size differences + this is 100% accurate to real result - no false positives and no missed streams.

[elit]

Alright so I made the script. Edit it for your game path, change ztool parameters as needed and run it. It need DD command for windows which you can get here:
http://www.chrysocome.net/dd.
Next you need 7zip, specifically 7z.exe and 7z.dll only, and ztool.

Place everything in one folder, including my attached script 19363 and you are good to go. This script will show time at the beginning and the end of work(not shown on the pic as I added it later) and it will show original as well as pre-processed size at the end:
19362

This script is slower than GFS as I found, probably because inefficient serial piping, but still "ok" and reliable. Unlike GFS, feel free to set up as many threads on ztool as you can to speed it up without reliability cost.

I recommend this for plz4 and plzo since they hang, you can keep using pzlib on GFS(with dynamic streams on) to get more or less same accuracy while being quicker. But if you want to have exact size that you would get using ztool, you can still use this for pzlib as well.

PS: since this doesnt utilize disk fully(like GFS do), you could have say 3 different batch script versions in the folder, one for pzlib, plz4 and plzo respectively and run all 3 of them at once.

[Razor12911]

Nice, problem is GFS finds more stuff, stuff that was incomplete in ztool. the leaked version for example cannot process crilayla streams, nor lzo streams from unreal engine games and so forth so yes, your alternative method works but to a certain extent

[elit]

Nice, problem is GFS finds more stuff, stuff that was incomplete in ztool. the leaked version for example cannot process crilayla streams, nor lzo streams from unreal engine games and so forth so yes, your alternative method works but to a certain extent

Indeed it look so. With that said though, to us regular people, the only relevant streams are the ones we can actually process. Even if GFS can find more, it is of no use if we lack the tool to process all of them. Thats why I think using ztool is(for us) better - to show us exactly what we get during actual packing.

[nik1967]

Why I did not show the size? And memory size for scanning?
http://img12.lostpic.net/2017/09/24/4e603e8e2af33a26c2c1aaf0b5557540.png

[KaktoR]

Same here.

For some it shows, for some not :D

[felice2011]

Why I did not show the size? And memory size for scanning?
http://img12.lostpic.net/2017/09/24/4e603e8e2af33a26c2c1aaf0b5557540.png

I always think about the same problem, the only solution.

http://fileforums.com/attachment.php?attachmentid=15271&d=1473292621

Enter the decimal point...change "," to "." and try.

[nik1967]

felice2011, thank you very much! Helped.

[KaktoR]

Thank you, that helped me too :D

[Webster17]

Thanks Razor for this tool, helps a lot. Anyways my question is can I use reflate with pzlib3 to recompress dynamic streams? Problem is I have a file whose size is 470mb after decompressing using unpacker becomes 700+mb. Also if I use GFS and set it to zlib+dynamicstreams it shows 700+ mb. But while recompressing using pzlib3 it only recompresses the file to
pzlib e -m2 -s file.ext -o file.pz
470mb=560mb(recompressed data) which after srep+lzma compression file becomes 305mb
and if I use this setting the
pzlib e -m2 -t25p -c128m -s -r1 file.ext
470mb=591mb after using srep+lzma compression file becomes 323mb.
Whats the problem can anybody help?

[78372]

^^ Try using higher chunk size(200+, you need to test to know), if the file contains large zlib streams

[Webster17]

Not working bro, btw the files inside are about 543, each file has less than 1-2 mb size. I have the file unpacker. Game File Scanner uses reflate dll's and so does pZLib3 but doesnt extract all, so is there any other way?

[78372]

Probably they are non-recoverable streams. Try with fast verification.

[Simorq]

pZLib3:c512m:x:s:r:t25p
ZTool=pzlib:m3:max

[panker1992]

use can even notify ztool on level

ztool=pzlib:high:ct75p:st:1:c250m
or for higher
ztool=pzlib:max:ct75p:st:1:c250m
or you can even go heavy metal like, it will check for level 9 deflate
ztool=pzlib:m3:r9:ct75p:st:1:c250m

Most efficient way to use ztool, have 1 thread to scan and all other threads to work

[ZakirAhmad]

any idea on why i'm getting worst ratio after srep+lolz...
i used the setting on Tomb Raider 2013 it mange to detect more zlib i get 15.7gb vs pzlib:m2 15.5gb but after applying srep+lolz i get 5.64gb, with pzlib:m2 i get 5.50gb

Bigger is not always better in case of razors precompressors. as they use xdelta to compute difference between streams, sometimes the diff is bigger which causes bigger size of precompressed file.
u should first scan the file with DS4ZLIB and try to match that size.
the tools like DS4ZLIB has been created by razor to avoid these types of errors with his tools.

[78372]

AFAIK Patchers aren't used for pZlib
But using reflate libs with pzlib when it's not needed will mess up ratio a bit

[doofoo24]

Bigger is not always better in case of razors precompressors. as they use xdelta to compute difference between streams, sometimes the diff is bigger which causes bigger size of precompressed file.
u should first scan the file with DS4ZLIB and try to match that size.
the tools like DS4ZLIB has been created by razor to avoid these types of errors with his tools.

NAH :D
i figured out the problem now i am getting 5.25gb on zlib and 400mb on bik video, final zise 5.64gb :p

[ZakirAhmad]

NAH :D
i figured out the problem now i am getting 5.25gb on zlib and 400mb on bik video, final zise 5.64gb :p

So, what was the problem and what was ur initial game size and what is now total final size.



U might have included .big sound files of game with (.big zlib game data) that is why u r getting.

[SAM2712]

When i scan .rpkg file with dynamic streams then zlib detects more than levels setting. So what precompressor is best for .rpkg file ? And if i using the zlib then how to set zlib for more detection. What's the best usage of zlib with 8 gb ram ? I am using this on hitman.

[IgorKolesnik]

How to understand what levels in Zlib are and how to apply them correctly in practice?

[yasitha]

How to understand what levels in Zlib are and how to apply them correctly in practice?

It doesn't matter..
if you found zlib streams,
just use xtool but before you do that

Do test run, i do it ll the time.

Get some random files from the game, witj zlib stream available.
Then try
xtool, zlib c32mb, c64mb, c128mb

when you change the chunk size game will be get batter ratio that happen with (the Evil Within) game.
So when you check all 3 of them youll know which one to use. If it's not different, then use c32mb.

[adammail14]

can somebody help me how to use this file scanner ? how to choose which config ?

what is chunk size and thread ? huhu i want to scan sims 4 and compress it, but i dont even know how scan :'(

[Masquerade]

can somebody help me how to use this file scanner ? how to choose which config ?

https://i.imgur.com/kTajQjA.png

And in the case of Sims 4 - check this post by KaktoR (https://fileforums.com/showpost.php?p=486172&postcount=1771) for an example of how to compress it.

[b2bcode]

The tool is tremendous but I don't know why ESET Internet Security is blocking it with showing suspicious activity dialog box. I can run it after temporary disabling antivirus. Anyone have similar issue?

[Masquerade]

Does dynamic stream detection indicate encrypted data?

[Razor12911]

it's the streams in dishonored 2 games and nothing special, I decided to name them dynamic streams because of the difference in headers.

[Masquerade]

it's the streams in dishonored 2 games and nothing special, I decided to name them dynamic streams because of the difference in headers.

I found these streams inside Automobilista 2 - is there a way for XTool to process these streams?

[Razor12911]

try the latest xtool in the 2020 thread using zlib+preflate and find out