cls-zstd v.25.03.04.rev.1:

- Updated zstd sources to version 1.5.7.
- Improved error handling system.
- Minor code optimizations.

cls-zlib v.25.03.09.rev.1
 

cls-lzav v.25.03.09.rev.1
 

OGG_File_Checker
Some small batch I wrote some weeks ago.

You can find the MESA tool here if you need it: https://www.patreon.com/posts/mesa-v-23-12-30-95531353

Example: Kingdome Come Deliverance 2, Music.pak
There are a total of 2816 OggS vorbis streams. 40 of them will cause OGGRE to crash or stuck in an infinite loop on decompression.

These bad files will be stored in a text file (Bad_Files.txt) and will be copied to _Bad_Files folder additionaly.

Additionaly this tool will create patch files (stored in Patch_Files folder) if you want to patch the bad files back to original files afterwards.


Checksum hashing is a bit slow especially if you have over 10k files (it will take a while). I didn't found a tool by now which works faster than xxhsum.

Thanks to César for a little help with checksum checking.

v2 changes: Subfolders are now supported. You can now copy/move entire game folder which includes OGG files and check them. Removed xdelta patch engine, only hdiff is supported now.

There are some example files included in v2.

CLS updates
 

cls-lizard v.25.03.16.rev.1

• Improved error handling system.
• Code cleanup and optimization.

-----------------------------

cls-unpackmp2 v.25.05.14.rev.1

• Updated and unified the CLS core.
• Optimized I/O performance.
• Code cleanup and general optimizations.

-----------------------------

cls-ttgd v.25.06.23.rev.1

• Implemented unified CLS core.
• Optimized I/O operations.
• Code cleanup and overall optimization.

P.S. You can get MESA here, as all my projects is free for everyone

cls-ue4d v.25.07.03.rev.1
 

Hello
Sometimes, conversion is weird...
I used cls-ue4d v.25.07.03.rev.1 on Sifu (last version)

I know it works pretty well because I uncrypted .pak and remove some data in stream.
With an hex editor i can see that the files is fully uncrypted.
(I can compare extracted files with quickbms, exactly the same in the uncrypted file)

With ued4t 99% uncrypted, with unreal xtool plugin around 98%
If I work on file to compare extracted files there is some diff
This makes tools like xtool missing some stream

By the way the think I do not understand, is with partial uncrypt like with unreal plugin, the final compress ratio is better

like for my tests :
cls-ue4d+srep+lolz => 12gb
xtool:mue4+srep+lolz => 10gb
ue4dt+srep+lolz= 9.96gb

Added new oodle library
- version 2.8.14 (lib2) from "CODMW3 (steam)"

Archiver WIP 1, A brand new archiver with much much work poored into it
 

This is a project i have been designing for the past year,

this is practically a brand new archiver, it has custom everything,

Brand new Header, like Freearc which with its utility has taken this forum for the past decade and more
this project's plan is to practically replace it.

I have added a ton of stuff, including the ability to use external algorithms and tools, and i have also included xtool 7.9 from Razor Github page into it as an internal means to precompress.

it comes with 2 modes,

Solid Block, and Blocked compression

you can rely on solid block to pack your games while a block compression can be used to maintain archives.

I plan to expand this with GUI and a DLL for Inno.

Microsoft is working on xtool-like tool (https://github.com/microsoft/preflate-rs)??

A rewrite of precomp-cpp for speed and modularity, supports stdin and stdout natively, https://github.com/nicolas-comerci/precomp-cpp

Its not M$. Its just a guy who calls himself ms. ;)

No, I'm pretty sure it's microsoft.

Interesting. Maybe u have right. ;)

Could someone write a "seemingly simple" CLS filter, a WIM format* (files store only compressed, aka hiddenly deduplicated) compressor with streaming functionality? (It would work exactly as if we were packing with the REP algorithm, as far as writing the ARC file is concerned, without temporary TEMP files) Many old games have a feature that a file exists in multiple copies, so we could avoid the classic "duped files removed before compression" and "Duping files... during the unpacking process" solutions.

*WIM-compression level: Store

This is available in 7-Zip, but a CLS filter would be a better solution. When preparing a set of WAV files of nearly 5GB in this way, it consumes only 10-12MB of memory, while compressing to 3.2GB. (This is how much less needs to be copied/written before running the next algorithm.) In other words, if we manually deleted all duplicate files, the starting state would be this much smaller before running the other algorithms. I know, this is not typical for today's games. In this case, REP/SREP is not a viable option if MSC were the next algorithm or, for example, XTOOL. (The size of the deflated data file would be smaller.)

We could also use ZPAQ, but it deduplicates at the file level, "with the -m0 or -m01 switches". In the example above, the prepared dataset will be around 3GB in size, but the final compression will be worse. (1.24GB vs. 1.35GB) With the -m0 switch, ~1.05GB, with -m01 it will consume ~150-170MB of memory during packaging. Compared to the 10-12MB used when 7-Zip/WIM compression.

Added new oodle library
- version 2.9.12 (lib3) from "Battlefield 6"
- version 2.8.7 from "Battlefield 2042"

Oodle
 

Oodle Version 2.9.14 Self compiled

CLS updates
 

Hi there. Just a quick note about updating my CLS. I'm too lazy to write a changelog, but you can find it on my Patreon. The changes are mostly aimed at improving stability.

I updated the tool to v2.
Subfolders are now supported. You can now copy/move entire game folder which includes OGG files and check them.
Removed xdelta patch engine, only hdiff is supported now.

I modified the source code and recompiled it with Visual Studio.
Changes: WavExtractor now extracts the files as WEM extension instead of WAV.

This tool is still usefull because MESA cannot extract some archives (Shegorat already knows about this problem).

Very fresh new competitor from others projects. (FreeArc Next, Diskpan, UCC, etc...)

See this: https://encode.su/threads/4480-SYC-M...mpression-Tool

Separately as a new post!

YadeWira released packJPG Multithreaded versions few hours ago! Windows x86/64 and Linux64 binaries in available!

https://encode.su/threads/4482-packJPG-Multi-threaded

Great.

I just tested a bit. It looks like -th0 is actually much slower than without this option.

-th6 is the same as -th0. Am I missunderstand this option?

Also you still have to use solid=0 option for freearc sadly.

Edit: I just saw this is the wrong toppic for such discusions.

Here’s what I replied on the encode.su forum: “To be honest, I haven’t tried it on FreeArc yet. It’s not recommended to use the -th option with a single file, since it yields worse results in terms of compression and decompression times. In any case, you should create an option that generates a package (.pjgp?) containing all the already-compressed .jpg files or something similar. I don’t know what you think."

oodlescan

a simple oodle stream scanner i made. 99% of the logic is taken from xtool source code. built with gcc on windows x64. the only reason i made this and difference between this and Razor12911's oo2scan_7_win64 is that this one uses chunk based data loading. oo2scan_7_win64 tries to occupy the entire size of the file on memory to store and scan it from there, in cases where the data is huge it would fail. oodlescan copies small chunks from the file to memory and scans from there, eliminating the memroy allocation issue. i will add the source code here after some polishing :).

edit: added source code

reserved

wem(un)pak

This is a more feature packed version of Masquerade's WemTool that i made few months ago to compress wem files from Silent Hill f. As it was made with golang, the binary sizes are huge. I didnt want to port this entire thing ito c, so today i added a feature to generate a batch script to decompress the files. Then you can run the batch script in parallel with prl or Mparallel. It follows same file naming standards as original WemTool. So you can use wempack to decompress WemTool compressed data. Attachments include source code.

• Separate compressor and decompressor.
• Ability to compare input and compressed sizes.
• Works recursively.
• Ability to specify compressor for hdiffz.
• Ability to generate a batch script to decompress.
• Better error handling and processing structure:
• compressor : ww2ogg > oggre_enc > oggre_dec > hdiffz
• decompressor : oggre_dec > hdiffz
• If any of the steps fails, the tool skips that file automatically. Temporary files are deleted in every step of processing, WemTool processes all files and then removes temps at the very end, leaving a huge mess and sometimes filling up the disk.

Credits:

• Masquerade for the guide
• Sisong for hpatchz/hdiffz
• hsc64 for ww2ogg
• Profrager for oggre

wrathma: This wempak tool, XP-compatible or Vista+ 32bit?? Some parts from Masquerade's WemTool, are yes running from XP.

the one above builds shouldnt work on 32bit systems, i have attached a 32bit builds for you. not sure if it will work correctly as i dont have a 32bit test system laying around. i replaced the 64bit hdiffz/hpatchz exe with 32bit ones, everything else is same.

Some small bug
Also it would be good to give some information on how long the process took.

Another bug in decode mode
Testing:
Surprised how fast this is compared to WemTool
Both tools are running with all cpu threads.

Edit:
Impressive
wempacker is 3x faster than wemtool

Suggestions:
Add -c option for hdiffz compression
Change skip -s option to skip files even if they are same size as input -> equal or greater then

Very nice tool wrathma and pleased someone has nulled the need for my bat2exe nightmare creation.

wrathma: Many questions!

wem-packer: Why are both "oggre_enc/oggre_dec" binaries needed? Also, how are they 9KB larger than the original files?
In the original "ww2ogg" package, there is another "packed_codebooks.bin" file. Could the lack of this cause problems in some cases?
There is a 64-bit version of this program on GitHub, I won't link it directly, find it and take a look, you'll get the hang of it!

There is also a "revorb" project, isn't that better than "ww2ogg"? Instead of the HDiff package, would xdelta3 be a worse solution? (it runs natively on XP too.) HDiffPatch XP issues from GitHub. (Two x86_64/i686 XP binaries found the posts.)

Is there a way to configure the program, especially "hdiffz", to use very little memory when creating a diff file? (It should comfortably fit within the 512MB upto 1-2GB memory limit.) It is true that when applying a diff file as a patch, in principle a lot of memory is required.

You're obviously asking, why XP?? If everything is true, around 2005 (or earlier?) games that have *.wem files were released and regarding these files, it would be possible to make the repacks publishable/archivable back to XP.

There's a bug in the 32bit version! The "hdiffz" binary file in it is 64bit!

Yesterday, I managed to make all the binaries run on XP. Although, a full native XP-compatible HDiff/patchz binaries would be better case/idea.

In the case of v4.8.0, the "hpathcz.exe", with the simple NT6.0 -> NT5.1 trick, runs smoothly, while the larger "hdiffz.exe" file, I had to tinker a bit more with CFF Explorer + 3 extra DLL files required, to run it.

In the case of v4.11+ versions, the latter situation is present for both files.

The only task that remains to be solved is to create an EXE file that can be run natively under XP by the GO-compiler. (Why does this require the "bcryptprimitives.dll" file?)

Added new oodle library
Some new libraries added. I don't know from where I have them since the last update. I am just collecting from games. I don't remember because I have not made notes about it.

i don't know if this is the right place to ask, sorry if my english is bad:

wich is the safest and stable cls for precomp right now? (with stdio support) and which
precomp version should i use?

which is the safest cls for srep? (or maybe should i use srep without it). what version is the most stable until today ? 2.2, 3.93, 4.90?


can i remake my old repacks with xtool and forget about pzlib (for zlib streams of course) or pzlib should be used for old games or something like that?

i used lolz 21a7, is there a new updated version?

inno setup 6xxx is stable with isdone dll?

many thanks!

^^
For stdio precomp, you can use this version: https://fileforums.com/showthread.php?t=105410

CLS-SREP by ProFrager is still the best cls-srep for decompression. Use srep.exe for compression. 3.93b is alright for most data.

pzlib is obsolete, use xtool.

lolz v22c4b is the latest lolz version.

Inno Setup 6 does work with isdone, yes.

wrathma: There are problems with this GOLang miracle! Does it need an AVX/AVX2-compatible processor? Or do the compiled binaries only work flawlessly on the given machine? What if I remove the GO compiler from the machine, will the binaries compiled until then break?

The GO compiler* used. (I also compiled THIS with it, on XP it prints out the Help nicely, that's all, when compiling (go.exe build source.go) it also prints out similar errors as what can be read below)

*Used last v1.26.2-1 i386 packages and separately compiled from sources. (This v1.24.4 from XP) from make EXEcutables.

The x32 binary you posted, writes these (under Win7 SP1 x64):
The x64 version also produces similar and longer errors. (Checked on two different Win7 x64 PCs.)
If you chain the compiled program to the given machine/Windows files, it is very bad and makes the given program and the compiler unreliable.
If the source code were rewritten to c/c++, how much better would it be? (With AI, I had a CPP produced, but the program is faulty, it could only serve as some guidance for porting to CPP.)

In its current state, the CPP file only produces "apparently" working programs after compilation!

So far, I have produced 3 different packages from it, but under XP it throws errors similar to the above, the program prints the prompt in the console window regardless. Under Win7 there are no more strange errors, but as it turned out, if you have to wait for IO errors, HDD or the CPU is too busy, then it will make errors, there is a possibility of making errors. On a laptop with a 2-core CPU, not foolproof, on an 8C/16T Xeon PC, it seems to be better. That is, it doesn't unpack the files, it doesn't run the "hpatchz.exe" file at first, or some other anomaly.

First version maked from Win7 via GO v1.26.2-1: Link
Second version maked from Win7 via GO v1.24.4, hopefully targeted WinXP: Link

Masquerade: Do you still have the BAT2EXE program that you used to create the "WemTool.exe" file? Would you create it again? I'm attaching the patched/replaced/updated files. That way it will be 100% compatible with XP!

i decided to pop some stuff after a long time, not only i updated my archiver to be a beast!

But i run a showcase to demonstrate the vast capabilities it can do!

i decided to run resident evil requiem precompression only

sorry, for the srep cls, are you talking about this one?:

https://krinkels.org/resources/cls-srep.261/

or are you talking about srep inside?

i cant donwload from krinkels, can you or anybody provide the md5 of the cls
so i can find it on my folders? (I surely have it, but I must be sure that it is the right one.)

i have lolz v22c4b, is everybody using it? is it stable?

thanks again

thanks for your detailed review KaktoR. the progress bar was shamelessly copied from stackoverflow and unfortunately it
doesnt work great with multithreaded operations. so i removed it and added a simple progress bar. should be fixed by now. also
added -c option for hdiffz. everything you pass after this will be passed directly to hdiffz -c-{here}. example -
both oggre encoder and decoder is required because sometimes doing oggre_enc and oggre_dec on a file gives different
files (crc error). but doing oggre_enc on the output of oggre_dec will always give same files. try this out yourself
on diffrent ogg files you will get it.
i have never used revord, i guess ww2ogg works so ill use it for now (rule of engineering, if it works dont touch it).

xdelta3 and hdiffz could be interchanged as they are basically doing the same thing. i have added a xdelta build for
you. keep in mind that xdelta produces a little bigger diff files compared to hdiffz. tho the difference is minor.
during my tests, the entire tool running on 20 threads didnt cross the 500 mb ram usage mark. stayed mostly on
380-400 mb of ram usage (wempak+hdiffz+oggre+ww2ogg). applying patch is easy while making patch is harder and takes
lot of ram. but when the files are small ram usage also goes down.

golang turned down support for windows xp long ago i forgot to mention that so this tool shouldnt really work on
windows xp. i dont even ship wemunpak on my repacks and neither should you. the binary is bloated with golang
bloat code. this is why i added a feature to make a batch file to run the entire decompression chain.
check the -n option. you can then ship the batch file with the tools (hpatchz+oggre) and a
parallel processor (prl or mparallel)

it doesnt need a avx/avx2 compatible processor to run. atleast the logic should
work on all processors. golang programs are compiled and they are standalone.
unlike python/nodejs you dont need a interpreter to be installed on the user end to run this.

honestly i dont see any reason to rewrite this tool into cpp/c. performance
difference would be minor or nonexistent, but this type of tool is not meant
to be coded in these low level languages. i have seen your cpp code, you mostly
copied my structure a to z but the encoding/decoding logic is not there. if you
complete this on c the binary sizes might be cut to half or even more. and it
would probably work on older systems. if you follow what i do, i pack the files
in a modern device and then ship the tools required with a batch file. well if
you want feel free to rewrite this tool in c/cpp.

Thanks for the update.

It's just a visual bug, so nothing to deal with on tryhard level.