Tools for Repacking

[Shegorat]

cls-zstd v.25.03.04.rev.1:

- Updated zstd sources to version 1.5.7.
- Improved error handling system.
- Minor code optimizations.

[Shegorat]

Changelog:

• Code cleanup and optimization.
• Improved error handling system.

[Shegorat]

Changelog:

• Updated LZAV sources to version 4.7.
• Improved error handling system.

[KaktoR]

OGG_File_Checker
Some small batch I wrote some weeks ago.

You can find the MESA tool here if you need it: https://www.patreon.com/posts/mesa-v-23-12-30-95531353

Example: Kingdome Come Deliverance 2, Music.pak
There are a total of 2816 OggS vorbis streams. 40 of them will cause OGGRE to crash or stuck in an infinite loop on decompression.

These bad files will be stored in a text file (Bad_Files.txt) and will be copied to _Bad_Files folder additionaly.

Additionaly this tool will create patch files (stored in Patch_Files folder) if you want to patch the bad files back to original files afterwards.


Checksum hashing is a bit slow especially if you have over 10k files (it will take a while). I didn't found a tool by now which works faster than xxhsum.

Thanks to César for a little help with checksum checking.

v2 changes: Subfolders are now supported. You can now copy/move entire game folder which includes OGG files and check them. Removed xdelta patch engine, only hdiff is supported now.

There are some example files included in v2.

[Shegorat]

cls-lizard v.25.03.16.rev.1

• Improved error handling system.
• Code cleanup and optimization.

-----------------------------

cls-unpackmp2 v.25.05.14.rev.1

• Updated and unified the CLS core.
• Optimized I/O performance.
• Code cleanup and general optimizations.

-----------------------------

cls-ttgd v.25.06.23.rev.1

• Implemented unified CLS core.
• Optimized I/O operations.
• Code cleanup and overall optimization.

P.S. You can get MESA here, as all my projects is free for everyone

[Shegorat]

Changelog:

• Integrated unified CLS core.
• Optimized I/O operations.
• Added and updated keys for approximately 50 games.

[wareck]

Hello
Sometimes, conversion is weird...
I used cls-ue4d v.25.07.03.rev.1 on Sifu (last version)

I know it works pretty well because I uncrypted .pak and remove some data in stream.
With an hex editor i can see that the files is fully uncrypted.
(I can compare extracted files with quickbms, exactly the same in the uncrypted file)

With ued4t 99% uncrypted, with unreal xtool plugin around 98%
If I work on file to compare extracted files there is some diff
This makes tools like xtool missing some stream

By the way the think I do not understand, is with partial uncrypt like with unreal plugin, the final compress ratio is better

like for my tests :
cls-ue4d+srep+lolz => 12gb
xtool:mue4+srep+lolz => 10gb
ue4dt+srep+lolz= 9.96gb

[KaktoR]

Added new oodle library
- version 2.8.14 (lib2) from "CODMW3 (steam)"

[panker1992]

This is a project i have been designing for the past year,

this is practically a brand new archiver, it has custom everything,

Brand new Header, like Freearc which with its utility has taken this forum for the past decade and more
this project's plan is to practically replace it.

I have added a ton of stuff, including the ability to use external algorithms and tools, and i have also included xtool 7.9 from Razor Github page into it as an internal means to precompress.

it comes with 2 modes,

Solid Block, and Blocked compression

you can rely on solid block to pack your games while a block compression can be used to maintain archives.

I plan to expand this with GUI and a DLL for Inno.

[futureking]

Microsoft is working on xtool-like tool (https://github.com/microsoft/preflate-rs)??

[futureking]

A rewrite of precomp-cpp for speed and modularity, supports stdin and stdout natively, https://github.com/nicolas-comerci/precomp-cpp

[shazzla]

Its not M$. Its just a guy who calls himself ms.

[futureking]

Originally Posted by shazzla

Its not M$. Its just a guy who calls himself ms.

No, I'm pretty sure it's microsoft.

[shazzla]

Interesting. Maybe u have right.

[kj911]

Could someone write a "seemingly simple" CLS filter, a WIM format* (files store only compressed, aka hiddenly deduplicated) compressor with streaming functionality? (It would work exactly as if we were packing with the REP algorithm, as far as writing the ARC file is concerned, without temporary TEMP files) Many old games have a feature that a file exists in multiple copies, so we could avoid the classic "duped files removed before compression" and "Duping files... during the unpacking process" solutions.

*WIM-compression level: Store

This is available in 7-Zip, but a CLS filter would be a better solution. When preparing a set of WAV files of nearly 5GB in this way, it consumes only 10-12MB of memory, while compressing to 3.2GB. (This is how much less needs to be copied/written before running the next algorithm.) In other words, if we manually deleted all duplicate files, the starting state would be this much smaller before running the other algorithms. I know, this is not typical for today's games. In this case, REP/SREP is not a viable option if MSC were the next algorithm or, for example, XTOOL. (The size of the deflated data file would be smaller.)

We could also use ZPAQ, but it deduplicates at the file level, "with the -m0 or -m01 switches". In the example above, the prepared dataset will be around 3GB in size, but the final compression will be worse. (1.24GB vs. 1.35GB) With the -m0 switch, ~1.05GB, with -m01 it will consume ~150-170MB of memory during packaging. Compared to the 10-12MB used when 7-Zip/WIM compression.