XTool 2020 (Main Project)

[elit]

@Razor12911,
Ok I already found a site that was capable to test my gigabit/hdd limits

Anyway, this link gives you 2 archives and FA bundle:
https://ufile.io/f/hjbf7

In FA bundle, I use freearc.exe(GUI). In bin\_EC\xtool\ directory are the relevant xtool files. Archives use xtool.exe, so you overwrite one of different versions to that. I believe I used v047 most of the time, but can't say for sure which one I used on those archives. However, none of them successfully unpacked under Win8.1.

You will be able to test 1F.arc to the almost end, which confirm that last chain with big chunk size set is the one problematic(see archive info from GUI). Inside 1R.arc is a single file and will bug out at the very beginning.

Please let me know what you think. Also if anyone will be able to unpack those without error, then it would confirm my suspicion about my Win8.1.

EDIT: Also, when creating new archive under Win8.1(same parameters), it will test and extract correctly. I tested it on "1R" content. Still don't believe this to be bitrot/disk corruption etc. More like different OS environment+HW is causing intolerance in between versions. Before I had Win7 and i5-4690K CPU. Now I have Win8.1 and Ryzen 5800X. This is not a good thing if the case, change of HW/OS should not affect decompression reliability.

[MAY/O]

Did you ever think about making a linux build?

[Razor12911]

I once did but it would cause me headaches as a Windows user and as such, the testing would most likely be relegated to Linux users and on this forum, there's really not that many. I mean even tests on Windows are too much for ask for at times

[MAY/O]

Originally Posted by Razor12911

I once did but it would cause me headaches as a Windows user and as such, the testing would most likely be relegated to Linux users and on this forum, there's really not that many. I mean even tests on Windows are too much for ask for at times

Now I got some more experience with compression, I am always open to help if you want.
But I totally get it if not! So don't worry

[felice2011]

Hi guys, I like to take this opportunity to say hello to old contacts who have known me here on Forum since 2011, and the new guys who are passionate about compressing and installing games. I'm glad that the forum is still active, even if not as I remembered...
A big hello to friend Zee, who still works flawlessly with his work to this day.
I've been away a lot for work, so I'm quite rusty on the subject.
I wanted to ask someone here in the thread if they had managed to implement the added function of srep in xtool, or at least if I understood correctly if could be done and how....
Added in last xtool "- srep64 executable considered in x64 build of xtool" - //- "-sp# - srep parameters (separate params with ":")". thankss

[elit]

With -mpng.. method, xtool may hang, remaining unresponsive. This probably happen with non-png data. For example if doing whole "Data" directory in "Total War: Warhammer 2".

With -mpng though, more than 20k new streams can be found in those .pack(s) and inflation of another 500mb can be achieved.
Test passed without hanging when -c32m was used, but I fear this is more like a workaround/cheating and I fear of using bigger chunks due to risk of not unpacking data sometime in the future again(as it was the case already as mentioned in my posts above).

pngbug.png

EDIT: I have narrowed down which specific file is causing the bug: https://www.udrop.com/7YDG/local_en_2.7z
It is replicable on my comp. Xtool069 fails/hang with reflate+zlib+png:d1 in its own GUI scanner, while it doesn't hang with c32m added.

[L33THAK0R]

Are there any switches to disable the 256kb minimum file size for the "Erase" functionality? I've got some tens of thousands of small ~20kb files that I'm trying to rip/erase but alas the minimum limit is making it a bit tricky.

[Masquerade]

The filter is for bytes, not kilobytes.

[L33THAK0R]

Originally Posted by Masquerade

The filter is for bytes, not kilobytes.

Huh, guess I must have messed something else up then!

[kj911]

Special questions: Hardly ways integrate in last xtool internal (not srep related!) deduplications functions adding to old xtool v0.12 versions?? Or make very tiny xtool compressors, named in xdedup or any names, in has ONE deduplications functions with stdio mode? (No zlib/reflate or many more functions..) Would I make sense?

[Razor12911]

Hi, the older versions of xtool and the current version have different code infrastructure and porting the deduplication code would be time consuming, also a standalone deduplication program wouldn't make sense simply because the xtool version is stream based meaning, it relies on whatever streams are detected, however if you just wanted deduplication in stdio mode, you can just use srep in -m0 mode rather than -m#f, not sure if that answers your request

[Razor12911]

Update available

Changes

- added ability to redirect base directory for plugins and libraries
- added restrictions to avoid errors with experimental codecs
- added optimize option to speed up the decoding process for zstd and oodle codecs
- added dictionary parameter for fast-lzma2
- added memory caching when decoding to alleviate speed bottleneck
- fixed bug with download feature for inputs in URL format
- fixed issues with exporting precompression database
- fixed issues with executable plugin support
- fixed issues advanced configuration based plugin support
- fixed potential decoding issue upon using plugin support functions
- fixed issues with deduplication feature
- fixed issues with jojpeg codec
- replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature
- replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios
- improved user interface
- improved oodle codec performance for 2.6.0+ libraries
- improved encoding speed when using internal codecs
- improved processing speed when depth is used
- removed fast lzma2 multi threaded decompression due to excessive memory requirements
- removed debugging information when using the patch function
- removed ability to toggle database feature and ability to export database files (now enabled by default)
- updated deduplication virtual memory allocation
- updated reflate codec to verify streams prone to data corruption

Notes
Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools.
Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added.

Speed when using -d# parameter has been improved as show below on the game Star Ocean The Divine Force's chunk_2.cpk file.

0.6.9

Compressed 1 file, 641,373,728 => 1,104,208,408 bytes. Ratio 172.16%
Compression time: cpu 0.53 sec/real 37.94 sec = 1%. Speed 16.90 mB/s

Tested 1 file, 1,104,208,408 => 641,373,728 bytes. Ratio 172.16%
Testing time: cpu 0.41 sec/real 73.39 sec = 1%. Speed 8.74 mB/s

0.7.0

Compressed 1 file, 641,373,728 => 1,104,208,460 bytes. Ratio 172.16%
Compression time: cpu 0.61 sec/real 30.71 sec = 2%. Speed 20.88 mB/s

Tested 1 file, 1,104,208,460 => 641,373,728 bytes. Ratio 172.16%
Testing time: cpu 0.45 sec/real 22.72 sec = 2%. Speed 28.23 mB/s

[dixen]

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

[Razor12911]

Originally Posted by dixen

Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this
<input_data_size> >> <Output_data_size>..

Streams: 58080 / 58080
Time: 00:01:27 (CPU 00:15:18)
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

I'm not sure what the question is, but the information shows up when you use the program w/o FA.

@shazzla

be mindful of base directory or -bd#, this allows xtool to relocate libraries and plugins (including xtl and ini files) from another directory

I added this because there were just too many files near xtool.exe, also if you want to compress games but rather than moving and deleting oo2core..., you can just change the base directory.

[kj911]

Originally Posted by Razor12911

Streams: 58080 / 58080
Time: 00:01:27 (CPU 00:15:18)
Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB]
Srep decompression memory: 45.0 MB [113 MB*]

Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma)

Final compressed size without srep?? Better or worser?

FLZMA2: This really? Small memory footprint + utilize extra more cores? (This feature available from native 7-Zip releases?)
Likelys the feature from LOLZ. More 4-12C/4-24T CPU's from in visible this ONE virtually core utiiized during in compression. (Faster speed in LDMF1 mode.)