XTool 2020 (Main Project)

[Razor12911]

it should be --srepmem=...

[elit]

I would like to bring to your attention an issue I am currently experiencing with xtool.

In the past, I compressed several games with it under Windows 7. Probably most used version by me back then was 047 and 053. I test every archive after creation and back then every one of them tested fine.

However about 2 months ago I switched to Windows 8.1 and I find myself facing decompression errors.
Specifically this almost exclusively relate to reflate(I had one zlib giving me error as well but not all). It is 100% reproducible and do not matter which xtool version I use(I keep ones I compressed anything with just in case). Reflate in archive = 100% fail. Any archive. Due to that I excluded disk issue.

Note that ztool with -cm0 work fine under Win8.1, as well as 32bit xtool-crilayla version. Problem is only xtool->reflate(maybe occasional zlib) from what I seen so far. Thankfully majority of my archives are ztool based due to me stubbornly sticking to it for so long but still quite few of new ones I made use xtool already. xtool doesn't have any fancy switches like -cm0, I tried -t1 to no avail.

I am currently trying all archives to make list of broken ones, I also already installed Win7 under Hyper-V late night and will try more on it later but don't know if environment will be 1:1, maybe xtool is having issues due to memory/exception handling of Win8.1.
Also, I did of course already tried Win7 compatibility(all relevant exe files in chain) as well as running as an admin(real one, enforced through group policy).

If anyone have an idea, I will appreciate it.

[elit]

Originally Posted by Gehrman

Use compatibility mode
Setup.bat

ECHO OFF
Reg.exe add "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe" /t REG_SZ /d "~ DPIUNAWARE WIN7RTM" /f
START Setup.exe
Reg Delete "HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers" /v "%~dp0Setup.exe"  /f
Exit

That's interesting, so it seems a known issue. Thank you, I will try later after I scan all my archives and report back. I hope this will really fix it.

[elit]

^Unfortunately, putting both freearc.exe and xtool.exe into registry like above did not help.err.png

EDIT: Not all reflate are broken, in fact many are fine. I am starting to see potential pattern but will confirm it later whether its case. Ones with the issue so far are the ones with custom settings, like chunk size -c32m etc., whereas all the good ones are just default -mxtool:reflate so far.

EDIT2: I see the pattern. This error affect both reflate and zlib, when high chunk size(>32mb) is used. This was working fine on Win7. Changing thread number doesn't seem to cut it. And I have same rig with enough memory(16G). I will try few more things but it seems as if Win8.1 is handling memory or relevant functions differently.

EDIT3: Windows 7 under Hyper-V did not help. I still believe archives are good and this is because I don't run under native.

[Gehrman]

@elit
Increase the size of the Windows page file (pagefile.sys).

[Razor12911]

could you try isolating the bug and send over the sample that causes xtool to bug out

[elit]

@Gehrman
I have 16G RAM + 16G swap. I also tried -t1 option. This is certainly not the case of RAM but also not of disk space.

@Razor12911
I have looked into it today and it does look like it may have to do with chunk size. All archives that bugged out had -c more than 32mb(64m, 128m), but those without any option or up to 32m were all fine. Best at this point will be for you to see yourself. I will upload 2 of them + FA whole pack that I use. With that you will be able to understand how I use it and try for yourself. Again this all worked under W7 native install. Also I am sure among those hundreds of archives where ztool was used, there had to be at least several with chunk size >32m and those passed fine(I tested my whole collection).
I will upload things somewhere and post the links by tomorrow.

[elit]

@Razor12911,
Ok I already found a site that was capable to test my gigabit/hdd limits

Anyway, this link gives you 2 archives and FA bundle:
https://ufile.io/f/hjbf7

In FA bundle, I use freearc.exe(GUI). In bin\_EC\xtool\ directory are the relevant xtool files. Archives use xtool.exe, so you overwrite one of different versions to that. I believe I used v047 most of the time, but can't say for sure which one I used on those archives. However, none of them successfully unpacked under Win8.1.

You will be able to test 1F.arc to the almost end, which confirm that last chain with big chunk size set is the one problematic(see archive info from GUI). Inside 1R.arc is a single file and will bug out at the very beginning.

Please let me know what you think. Also if anyone will be able to unpack those without error, then it would confirm my suspicion about my Win8.1.

EDIT: Also, when creating new archive under Win8.1(same parameters), it will test and extract correctly. I tested it on "1R" content. Still don't believe this to be bitrot/disk corruption etc. More like different OS environment+HW is causing intolerance in between versions. Before I had Win7 and i5-4690K CPU. Now I have Win8.1 and Ryzen 5800X. This is not a good thing if the case, change of HW/OS should not affect decompression reliability.

[MAY/O]

Did you ever think about making a linux build?

[Razor12911]

I once did but it would cause me headaches as a Windows user and as such, the testing would most likely be relegated to Linux users and on this forum, there's really not that many. I mean even tests on Windows are too much for ask for at times

[elit]

With -mpng.. method, xtool may hang, remaining unresponsive. This probably happen with non-png data. For example if doing whole "Data" directory in "Total War: Warhammer 2".

With -mpng though, more than 20k new streams can be found in those .pack(s) and inflation of another 500mb can be achieved.
Test passed without hanging when -c32m was used, but I fear this is more like a workaround/cheating and I fear of using bigger chunks due to risk of not unpacking data sometime in the future again(as it was the case already as mentioned in my posts above).

pngbug.png

EDIT: I have narrowed down which specific file is causing the bug: https://www.udrop.com/7YDG/local_en_2.7z
It is replicable on my comp. Xtool069 fails/hang with reflate+zlib+png:d1 in its own GUI scanner, while it doesn't hang with c32m added.

[L33THAK0R]

Are there any switches to disable the 256kb minimum file size for the "Erase" functionality? I've got some tens of thousands of small ~20kb files that I'm trying to rip/erase but alas the minimum limit is making it a bit tricky.

[Masquerade]

The filter is for bytes, not kilobytes.

[L33THAK0R]

Originally Posted by Masquerade

The filter is for bytes, not kilobytes.

Huh, guess I must have messed something else up then!

[kj911]

Special questions: Hardly ways integrate in last xtool internal (not srep related!) deduplications functions adding to old xtool v0.12 versions?? Or make very tiny xtool compressors, named in xdedup or any names, in has ONE deduplications functions with stdio mode? (No zlib/reflate or many more functions..) Would I make sense?