#601
|
||||
|
||||
Hi, the older versions of xtool and the current version have different code infrastructure and porting the deduplication code would be time consuming, also a standalone deduplication program wouldn't make sense simply because the xtool version is stream based meaning, it relies on whatever streams are detected, however if you just wanted deduplication in stdio mode, you can just use srep in -m0 mode rather than -m#f, not sure if that answers your request
|
Sponsored Links |
#602
|
||||
|
||||
Update available
Changes - added ability to redirect base directory for plugins and libraries - added restrictions to avoid errors with experimental codecs - added optimize option to speed up the decoding process for zstd and oodle codecs - added dictionary parameter for fast-lzma2 - added memory caching when decoding to alleviate speed bottleneck - fixed bug with download feature for inputs in URL format - fixed issues with exporting precompression database - fixed issues with executable plugin support - fixed issues advanced configuration based plugin support - fixed potential decoding issue upon using plugin support functions - fixed issues with deduplication feature - fixed issues with jojpeg codec - replaced crc32c with xxh3_128 to reduce collisions when using the database and deduplication feature - replaced memory manager with FastMM4-AVX to improve scaling in multi threaded scenarios - improved user interface - improved oodle codec performance for 2.6.0+ libraries - improved encoding speed when using internal codecs - improved processing speed when depth is used - removed fast lzma2 multi threaded decompression due to excessive memory requirements - removed debugging information when using the patch function - removed ability to toggle database feature and ability to export database files (now enabled by default) - updated deduplication virtual memory allocation - updated reflate codec to verify streams prone to data corruption Notes Database files created using old tools (ucas database, dunia2 database etc) are not supported in this version, wait for updates for these tools. Reflate may be slightly slower compared to previous versions and that's because verification on suspected streams that may cause crc errors has been added. Speed when using -d# parameter has been improved as show below on the game Star Ocean The Divine Force's chunk_2.cpk file. 0.6.9 Code:
Compressed 1 file, 641,373,728 => 1,104,208,408 bytes. Ratio 172.16% Compression time: cpu 0.53 sec/real 37.94 sec = 1%. Speed 16.90 mB/s Tested 1 file, 1,104,208,408 => 641,373,728 bytes. Ratio 172.16% Testing time: cpu 0.41 sec/real 73.39 sec = 1%. Speed 8.74 mB/s Code:
Compressed 1 file, 641,373,728 => 1,104,208,460 bytes. Ratio 172.16% Compression time: cpu 0.61 sec/real 30.71 sec = 2%. Speed 20.88 mB/s Tested 1 file, 1,104,208,460 => 641,373,728 bytes. Ratio 172.16% Testing time: cpu 0.45 sec/real 22.72 sec = 2%. Speed 28.23 mB/s Last edited by Razor12911; 27-04-2023 at 22:38. Reason: Hotfix uploaded |
The Following 18 Users Say Thank You to Razor12911 For This Useful Post: | ||
BKR-TN (05-05-2023), BLACKFIRE69 (28-04-2023), Cesar82 (27-04-2023), dixen (26-04-2023), Gehrman (26-04-2023), hdneo (26-04-2023), KaktoR (27-04-2023), kj911 (28-04-2023), kuyhaa (28-04-2023), L0v3craft (27-04-2023), L33THAK0R (26-04-2023), Masquerade (26-04-2023), Mini (26-04-2023), ScOOt3r (27-04-2023), seryogakms (27-04-2023), shazzla (28-04-2023), vint56 (27-04-2023), Wanterlude (27-04-2023) |
#603
|
|||
|
|||
Just a tip, nothing more..
What about the progress status for -dd3 and fast-lzma? For example this <input_data_size> >> <Output_data_size>.. |
#604
|
|||
|
|||
Something is wrong with Xtool 07 hotfix.
It doesnt catch up any stream on Snowrunner's EDITOR.PAK ,while Xtool 069 does ! (069 fails at the end on this file with reflate ,zlib works fine !) See below : ...... Edit #2 : Ohh ,maaan! ALL DLL's must be in the libraries folder ! Not only the external ones ,like unreal ,unity,etc... Last edited by shazzla; 28-04-2023 at 06:43. |
#605
|
||||
|
||||
Quote:
Code:
Streams: 58080 / 58080 Time: 00:01:27 (CPU 00:15:18) Duplicates: 4893 (18.2 MB) [26.2 MB >> 265 MB] Srep decompression memory: 45.0 MB [113 MB*] Size: 1.09 GB (original) >> 2.32 GB (zlib) >> 2.06 GB (dedup) >> 1.75 GB (srep) >> 915 MB (flzma) @shazzla be mindful of base directory or -bd#, this allows xtool to relocate libraries and plugins (including xtl and ini files) from another directory I added this because there were just too many files near xtool.exe, also if you want to compress games but rather than moving and deleting oo2core..., you can just change the base directory. Last edited by Razor12911; 28-04-2023 at 06:57. |
The Following User Says Thank You to Razor12911 For This Useful Post: | ||
shazzla (28-04-2023) |
#606
|
|||
|
|||
This "base directory or -bd#..." is not so clear...
How to use/set -bd# ? |
#607
|
||||
|
||||
Let's say you want want to move all libraries and plugins to a folder called "libs" which is near xtool.exe, you specify base directory as -bdlibs or -bd.\libs. or you can just leave it empty to make xtool work the same as before.
|
#608
|
|||
|
|||
Sorry for dumb question, but how get this info in xtool?
Quote:
|
#610
|
||||
|
||||
The information is always visible and verbose option isn't necessary because that will force xtool to use 1 thread, whether you use GUI or CLI. It's always there.
|
#611
|
||||
|
||||
There's a feature I'd like to point out which was briefly added in 0.6.8 but wasn't finished until this current release and that is on the fly compression, basically if you wanted to quickly find out the theoretical compressed size of a game without consuming disk space.
Reasons you may want to use this feature: + If you just wanted to find out the compressed size + If a game uses kraken and wanted to see if it was worth it to compress it at all + If the game is big and you don't have space for compression + If you don't want to reduce lifespan on SSD Code:
XTool is created by Razor12911 Streams: 1415335 / 1415335 Time: 00:01:21 (CPU 00:04:03) Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB] Size: 15.3 GB >> 29.0 GB >> 13.0 GB Done!!! Code:
Streams: 1415335 / 1415335 Time: 00:04:36 (CPU 00:34:05) Duplicates: 1134306 (1.99 GB) [7.11 GB >> 16.1 GB] Srep decompression memory: 751 MB [5.15 GB*] Size: 15.3 GB >> 29.0 GB >> 13.0 GB >> 9.03 GB >> 4.65 GB Done!!! _________________________________ The other thing, as you know how much I tried to make xtool to perform as fast as possible, the thing that always bothered me was that lzma was slow and that's why I added fast-lzma2. I noticed something interesting about this compression library in the sense that it's able to increase the number of threads used for compression without increasing memory usage. Here's a few results 4x4:lzma (via fazip) (b128mb:t6), ultra, 64mb dictionary, 12 threads used 2781 MB ram Code:
Compressed 288 files, 16,391,993,193 => 4,869,896,897 bytes. Ratio 29.71% Compression time: cpu 13.50 sec/real 662.85 sec = 2%. Speed 24.73 mB/s Code:
Compressed 288 files, 16,391,993,193 => 4,929,451,310 bytes. Ratio 30.07% Compression time: cpu 11.84 sec/real 425.39 sec = 3%. Speed 38.53 mB/s |
#612
|
|||
|
|||
Thank you Razor12911 for this fantastic tool...have one question, if it is not too much trouble, can you please consider naming x64 srep.exe (since xtool is hardcoded using this exact name) like Bulat original named it as srep64.exe?
I'm using original naming for many years, I need 32bit srep.exe for some compatibility reasons of my own...and your renaming 64bit srep64.exe as srep.exe is causing problems. Maybe at least consider making xtool using srep64.exe name first, if found, then if not found use srep.exe or whatever srep name you consider acceptable. Thanks in advance, best regards Last edited by infovs; 30-04-2023 at 01:26. |
#613
|
|||
|
|||
Quote:
FLZMA2: This really? Small memory footprint + utilize extra more cores? (This feature available from native 7-Zip releases?) Likelys the feature from LOLZ. More 4-12C/4-24T CPU's from in visible this ONE virtually core utiiized during in compression. (Faster speed in LDMF1 mode.) |
#614
|
||||
|
||||
Guys, does anyone know how to fix this?
Game: A Plague Tale Requiem Files: *.DPC Xtool: last ver 0.7.0 hotfix Problem: after unpacking 1 file (28.9gb) xtool goes beyond its allocated memory. (x4) The picture shows the information from windows resource monitoring. (I have 16 GB RAM) Code:
[External compressor:xtool] header = 0 unpackcmd = "xtool.exe" decode -t100p -dm20p -sm20p - - <stdin> <stdout> [External compressor:xtool] header = 0 unpackcmd = "xtool.exe" decode -t100p -sm20p --mem=20p - - <stdin> <stdout> arc.ini: Code:
[External compressor:xtool] header = 0 default = -c128mb -t100p packcmd = xtool.exe precomp -mzlib -d1 -db -dd5 - - <stdin> <stdout> upd. (14 may): preflate has the same problem | xtool 0.6.9 - same ----------------------- upd. (15 may): fixed by 2nd xtool and -dm20p -sm20p parameters. I think the problem was that I used 1 xtool for both zlib and dedup ----------------------- Last edited by Wanterlude; 15-05-2023 at 09:04. |
#615
|
||||
|
||||
Update available
Changes - fixed issues with fast-lzma2 being unable to set correct compression level - updated deflate stream scanner |
The Following 14 Users Say Thank You to Razor12911 For This Useful Post: | ||
Cesar82 (22-05-2023), cradames (22-05-2023), dixen (22-05-2023), ffmla (31-05-2023), Gehrman (23-05-2023), hdneo (22-05-2023), kuyhaa (22-05-2023), L0v3craft (26-05-2023), Masquerade (22-05-2023), Pantsi (25-05-2023), ScOOt3r (22-05-2023), shazzla (22-05-2023), vint56 (22-05-2023), Wanterlude (22-05-2023) |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[Dev]XTool | Razor12911 | Conversion Tutorials | 180 | 23-10-2020 07:26 |
Project Cars Digital Edition (3xDVD5) (srep+lzma) | GTX590 | PC Games - CD/DVD Conversions | 10 | 28-08-2017 09:34 |
Project IGI Anthology 1xCD700 CIUV2 2039 | mausschieber | PC Games - CD/DVD Conversions | 0 | 24-07-2017 16:12 |
Space Channel 5 Part 2 Translation Project | Christuserloeser | DC Games | 0 | 21-06-2004 19:16 |