Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #646  
Old 15-07-2023, 06:33
kj911 kj911 is offline
Registered User
 
Join Date: Apr 2010
Location: world
Posts: 182
Thanks: 132
Thanked 63 Times in 44 Posts
kj911 is on a distinguished road
infovs: Please testing more version older dlls...

Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.

This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.
Reply With Quote
Sponsored Links
  #647  
Old 15-07-2023, 07:05
infovs infovs is offline
Registered User
 
Join Date: Feb 2005
Location: Home
Posts: 44
Thanks: 2
Thanked 10 Times in 7 Posts
infovs is on a distinguished road
Quote:
Originally Posted by kj911 View Post
infovs: Please testing more version older dlls...
Example: 1.2.11/172 544byte and 104 448byte releases. And older 1.2.8/1.2.3 and more.
This (p)reflate library impact or not decompression speed? hif/raw dlls and preflate dll present it unpacker files folders.
I tested all zlibwapi.dll versions that I could find since ztool..Razor12911 version from 2020-07-18 (535552 bytes) was the fastest at decompression. I also did my own compile of zlibwapi.dll 1.2.11 with MS VC 2010 and also 1.2.13 with MS VC 2010/2015/2022 -> VC 2022 1.2.13 compile produced best result but Razor12911 version was still fastest at unpacking, although not by much, speeds were pretty close. Actually only this latest 2023-07-11/512000 bytes version speed drop was noticeable for me, that's why I posted my observation.
Also, if you use MS VC and want your compile to be XP compatible you are pretty much forced to use VC 2010..yes, you can use XP platform toolset on some later MS VC versions but I had some issues with it. Never compiled with MinGW though so I can't comment on that.

Last edited by infovs; 15-07-2023 at 07:22.
Reply With Quote
The Following User Says Thank You to infovs For This Useful Post:
kj911 (16-07-2023)
  #648  
Old 15-07-2023, 07:22
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,746
Thanks: 2,141
Thanked 11,098 Times in 2,295 Posts
Razor12911 is on a distinguished road
Code:
vc22_vista_1.2.13: 24688ms 24687ms 24750ms 24750ms 24782ms
vc17_winxp_1.2.13: 24375ms 24172ms 24156ms 24266ms 24359ms
rad_studio_1.2.13: 24359ms 24219ms 24203ms 24265ms 24313ms

vc19_vista_1.2.11: 21734ms 21672ms 21610ms 21843ms 21578ms
vc22_vista_1.2.11: 21391ms 21359ms 21391ms 21343ms 21282ms
vc17_winxp_1.2.11: 21453ms 21469ms 21359ms 21328ms 21360ms

vc22_vista_1.2.8: 21282ms 21281ms 21266ms 21281ms 21281ms

zlib-ng_1.2.13: 7843ms 7797ms 7781ms 7813ms 7828ms
Quote:
Originally Posted by infovs View Post
Don't really know what you did with 1.2.11/2020-07-18 compile because that was clearly the fastest zlibwapi.dll, at least for me. Yes, I know it is older 1.2.11 version and maybe missing some updated code but I never encountered any problems with it
I didn't do anything to 1.2.11 to make it fast, 1.2.13 itself is the one that is slow as you can see with the benchmarks, winxp compile has little to no impact on performance.

There's an issue opened on github regards to the speed which can be found here.
https://github.com/madler/zlib/issues/820

There was also a security vulnerability discovered in 1.2.11 and I don't know if patching this issue is what made zlib slower or not moving forward.
https://www.cybersecurity-help.cz/vdb/gnu/zlib/1.2.11/

The beauty of xtool is that the user can swap libraries if they prefer to use their own, there's also zlib-ng (utilizes modern CPU features) benchmarks added, as you can see it is faster however I doubt it produces matching crc as standard zlib but good to know nonetheless.
Reply With Quote
The Following 2 Users Say Thank You to Razor12911 For This Useful Post:
hdneo (16-07-2023), kj911 (16-07-2023)
  #649  
Old 29-07-2023, 15:56
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,746
Thanks: 2,141
Thanked 11,098 Times in 2,295 Posts
Razor12911 is on a distinguished road
Update available

Changes

- added universal scanner for DirectStorage gdeflate streams
- added the use of gpu for caching and virtual memory purposes
- updated depthing feature to improve stream detection when used by plugins

Notes
With the release of the game ratchet & clank: rift apart which used direct storage's gdeflate compression within its game files, I thought... hm, why not just add a scanner for these streams and so I did after investigating the stream structure (was easy to figure out actually). Use via -mgdeflate.

Will games use direct storage? I don't know but if they do then at least xtool has support for the streams used in the games, I just added it so that repackers don't flood me with DMs when they have troubles repacking games, maybe Starfield might use this...

After several tests, the gpu feature can now be used by the general public, use via -g#
Reply With Quote
The Following 14 Users Say Thank You to Razor12911 For This Useful Post:
-tara (29-07-2023), Cesar82 (29-07-2023), dixen (30-07-2023), elit (11-10-2023), Gehrman (30-07-2023), hdneo (31-07-2023), KaktoR (30-07-2023), kuyhaa (29-07-2023), L0v3craft (30-07-2023), L33THAK0R (30-07-2023), Masquerade (30-07-2023), ScOOt3r (29-07-2023), shazzla (30-07-2023), Wanterlude (30-07-2023)
  #650  
Old 31-07-2023, 02:29
L33THAK0R's Avatar
L33THAK0R L33THAK0R is offline
Registered User
 
Join Date: Feb 2021
Location: Saudi Arabia
Posts: 352
Thanks: 119
Thanked 59 Times in 38 Posts
L33THAK0R is on a distinguished road
Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?
Reply With Quote
  #651  
Old 01-08-2023, 08:22
daveyrob daveyrob is offline
Registered User
 
Join Date: Oct 2021
Location: In My House
Posts: 7
Thanks: 3
Thanked 2 Times in 2 Posts
daveyrob is on a distinguished road
Quote:
Originally Posted by L33THAK0R View Post
Is there any way to use XTool to erase compressed streams? I'm currently looking at the various Quantic Dream PC ports (i.e. Heavy Rain, Beyond Two Souls, Detroit Become Human) and they have the localized audio compressed using deflate/zlib, I'm not sure if its already possible with XTool but can a database be generated, and then an input folder in addition to a generated database be fed as parameters to XTool to target and blank out these streams?
xtool erase <folder> <archive> <database>
Reply With Quote
The Following User Says Thank You to daveyrob For This Useful Post:
Razor12911 (08-08-2023)
  #652  
Old 08-08-2023, 17:28
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,746
Thanks: 2,141
Thanked 11,098 Times in 2,295 Posts
Razor12911 is on a distinguished road
Update available

Changes

- fixed an issue where xtool would crash on Windows XP and other systems upon closure
- memory optimisations
- improvements with database "based" plugins
Reply With Quote
The Following 11 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (08-08-2023), dixen (09-08-2023), Gehrman (09-08-2023), KaktoR (09-08-2023), kj911 (10-08-2023), L0v3craft (10-08-2023), Lord.Freddy (08-08-2023), NERV (09-08-2023), Pantsi (09-08-2023), shazzla (08-08-2023), Wanterlude (09-08-2023)
  #653  
Old 09-08-2023, 13:07
infovs infovs is offline
Registered User
 
Join Date: Feb 2005
Location: Home
Posts: 44
Thanks: 2
Thanked 10 Times in 7 Posts
infovs is on a distinguished road
Thank you Razor12911, also..is there a reason not mentioning there is also new CLS version of xtool in your 0.7.8 update? Not stable/recommended yet?
Reply With Quote
  #654  
Old 09-08-2023, 13:19
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 1,167
Thanks: 284
Thanked 1,358 Times in 615 Posts
Masquerade is on a distinguished road
^^
Run some tests and see what you find. That's what I'm gonna do when I return to my PC.

Please note that FreeArc is an x86 program, so in order to use the x64 CLS variant, you will first need to use some kind of wrapper such as wrapcls by 78372 or FAZip (not FAZip32).

It's gonna be a lot of fun to play with. A cls is what I've been secretly wishing for, thanks Razor
Reply With Quote
  #655  
Old 06-09-2023, 05:23
shazzla shazzla is offline
Registered User
 
Join Date: Nov 2010
Location: Hunnia
Posts: 273
Thanks: 501
Thanked 94 Times in 71 Posts
shazzla is on a distinguished road
@Razor :

Maybe an XTool 0.7.8 bug ,using FLAC. Why is that increased memory consumption ?

Compression : -mflac:l8:c128mb , 1 thd ,approx. 800 MB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

Compression : -mflac:l8:c128mb , 4 thds ,approx. 1.8 GB RAM usage
Decompression : 1 thd ,approx. 180 MB RAM usage
Decompression : 2 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)
Decompression : 4 thds ,approx. 2 GB RAM usage (Starts with 3 GB ,fastly drops to ~2 GB)

All decompression settings used % to set the number of threads. Using T4 instead of t100p cannot decompress the archive :
ERROR: general (de)compression error in flac:l8:c128mb

Maybe other cases affected. Dont know.

i5-6600
Reply With Quote
  #656  
Old 06-09-2023, 11:56
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,746
Thanks: 2,141
Thanked 11,098 Times in 2,295 Posts
Razor12911 is on a distinguished road
There's two different code sets for decoding in xtool, one is for multi threading which caters for speed at a cost of increased memory usage because the streams have to be reorganised before they are written to disk and as such, high memory usage because they are stored in memory first. The other code is when you have set 1 thread of if a block has 1 stream in it, for this as soon as a stream is processed, it is written to disk hence why the reduced memory usage in -t1.

For flac, I'd suggest using -t1 for decoding simply because flac is decompressing when installing unlike other codecs where they require multi threading to be fast as they are doing the opposite (compressing) during installation.

A small test on some audio files

Code:
Tested 39 files, 745,011,186 => 1,379,878,320 bytes. Ratio 53.99%
Testing time: cpu 1.08 sec/real 4.68 sec = 23%. Speed 295.10 mB/s
Using 12 threads, used 1.23 GB ram

Code:
Tested 39 files, 745,011,186 => 1,379,878,320 bytes. Ratio 53.99%
Testing time: cpu 0.97 sec/real 10.18 sec = 10%. Speed 135.58 mB/s
Using 1 thread, used 185 MB ram.

The single threaded speed is reasonable while using 12x number of threads only gave 2.2x speed benefits while using nearly 7x memory. Obviously due to IO bottleneck show that it's better to just use 1 thread.
Reply With Quote
The Following User Says Thank You to Razor12911 For This Useful Post:
shazzla (06-09-2023)
  #657  
Old 08-09-2023, 07:17
infovs infovs is offline
Registered User
 
Join Date: Feb 2005
Location: Home
Posts: 44
Thanks: 2
Thanked 10 Times in 7 Posts
infovs is on a distinguished road
Hi Razor12911, I know similar question has been posted before but can you please re-consider renaming hardcoded srep.exe call in x64 xtool version to original, srep64.exe ? Since you made xtool available for both x86 and x64 versions I started using both when appropriate (making XP compatible installers for games that can be played on XP...) and hardcoded srep.exe is giving me headaches because both x86 and x64 are executing srep.exe. x64 xtool version should be executing srep64.exe IMHO
Anyway, thanks again for your xtool, best regards
Reply With Quote
  #658  
Old 10-09-2023, 07:32
Razor12911's Avatar
Razor12911 Razor12911 is offline
Noob
 
Join Date: Jul 2012
Location: South Africa
Posts: 3,746
Thanks: 2,141
Thanked 11,098 Times in 2,295 Posts
Razor12911 is on a distinguished road
Update available

Changes

- added recompress streams feature
- added reassign streams feature
- added dummy codec
- added data transfer buffer for srep when dd# is used
- added memory checks to ensure memory usage does not run wild
- exectuable plugins (via xtool.ini) are no longer required when decoding if they were never used
- internal stream deduplication now reports speed and memory usage benefits
- configuration based plugins can now have multiple names (if multiple games use the same configuration)
- fixed xtool crash if an incorect path for plugins was provided
- fixed lz4/lz4hc codec bug when used directly without any plugin
- fixed an issue where using fast-lzma2 compression would crash when decoding
- fixed an issue in DirectStorage gdeflate codec
- fixed an issue with execute command in stdio mode
- deduplication memory requirements improved
- memory optimisations
- reduced memory requirements for large streams that require patching
- removed ability to inject libraries to exe (buggy)
- removed patch command (uses too much memory)
- removed archive command (no one uses it)
- updated oodle scanner
- updated lz4/lz4hc universal scanner
- updated lz4/lz4hc/lz4f codecs
- updated zlib codec
- updated zstd codec
- replaced gpu caching feature with normal system memory cache (it doesn't work well on AMD gpus), use -p#
- replaced xdelta3 with zstd patching engine
- srep64 executable considered in x64 build of xtool

Notes

First and foremost, huge thanks to KaktoR for helping with this update, he was full of energy when we started but I think I broke him towards the end...
kekw.png

This update was supposed to be released a long time ago, but he would find more bugs hence the long list of changes.

I won't say much other than notable changes, xdelta was replaced by zstd patching engine (yes... zstd can make patches too, shocking I know...), reason for replacement xdelta gives me less options coding wise to work with in terms of memory management.

Injecting libraries feature also removed as some libraries when injected completely stop working or worse, cause xtool to crash.

I've quietly added cls plugin in the previous update and did not tell anyone about it simply because it was never tested, for this update It was tested and it works so long as you never used fast-lzma2 in any of the archives when using it with installations created with inno setup as it would simply cause a crash.

I've left 0.7.8 on the main post in case there are issues with this update even after extensive testing.
Reply With Quote
The Following 16 Users Say Thank You to Razor12911 For This Useful Post:
Cesar82 (10-09-2023), dixen (10-09-2023), elit (30-09-2023), felice2011 (25-09-2023), hdneo (10-09-2023), KaktoR (10-09-2023), kj911 (10-09-2023), L0v3craft (11-09-2023), Lord.Freddy (10-09-2023), Masquerade (10-09-2023), mausschieber (10-09-2023), NERV (15-09-2023), Pantsi (11-09-2023), ScOOt3r (10-09-2023), shazzla (10-09-2023), Wanterlude (10-09-2023)
  #659  
Old 12-09-2023, 23:27
L33THAK0R's Avatar
L33THAK0R L33THAK0R is offline
Registered User
 
Join Date: Feb 2021
Location: Saudi Arabia
Posts: 352
Thanks: 119
Thanked 59 Times in 38 Posts
L33THAK0R is on a distinguished road
Very embarrassed to ask such a simple question regarding database creation, but I can't seem to wrap my head around it. I've got some assets I'm attempting to rip some assets that are compressed inside an archive using deflate.

To do this I'm pretty confident in saying xtool just needs a database to reference, with the command looking like

Code:
xtool erase <folder> <archive> <database>
as a user pointed out to me a while ago.

Where I get stuck is creating the database, the title in question is "The Wanderer" a UE4 title (unencrypted) with a singular ~30GB ".pak" archive containing said assets, I've extracted the assets I'm interested in ripping to their own folder and have so far tried using xtool 0.7.9's GUI to generate the database since I'm not too confident the syntax I'm using is correct when running xtool from the command line.

Here is how I've setup the database creation though I'll admit I'm unsure of whether I've got the "Input" and "Source" correct.



However all I get from this configuration is the following message, which hangs for a few minutes despite some disk activity as per task manager, and then indicates the process is completed however no database file is created.



Very unsure and confused about what parts I'm mucking up or whether XTool just has a limitation on the size of inputs that can be processed.
Reply With Quote
  #660  
Old 13-09-2023, 01:13
Masquerade's Avatar
Masquerade Masquerade is offline
Registered User
 
Join Date: Jan 2020
Location: Monte d'Or
Posts: 1,167
Thanks: 284
Thanked 1,358 Times in 615 Posts
Masquerade is on a distinguished road
Don't use the GUI. In that screenshot, you are using the wrong database generator and your inputs+outputs need to be reversed.

And you cannot rip compressed data. It must appear in plaintext in order to be erased by xtool.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Dev]XTool Razor12911 Conversion Tutorials 180 23-10-2020 07:26
Project Cars Digital Edition (3xDVD5) (srep+lzma) GTX590 PC Games - CD/DVD Conversions 10 28-08-2017 09:34
Project IGI Anthology 1xCD700 CIUV2 2039 mausschieber PC Games - CD/DVD Conversions 0 24-07-2017 16:12
Space Channel 5 Part 2 Translation Project Christuserloeser DC Games 0 21-06-2004 19:16



All times are GMT -7. The time now is 16:57.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com