Tools for Repacking

[panker1992]

Oodle Version 2.9.14 Self compiled

[Shegorat]

Hi there. Just a quick note about updating my CLS. I'm too lazy to write a changelog, but you can find it on my Patreon. The changes are mostly aimed at improving stability.

[kj911]

Very fresh new competitor from others projects. (FreeArc Next, Diskpan, UCC, etc...)

SYC - Modular Compression Tool

SYC is not a compressor — it's a modular compression orchestrator that chains external tools (zstd, srep, xprecomp, zpaqfranz and more) into a single pipeline, producing .syc archives. Think of it as a conductor: it manages the workflow, archive format, encryption, and progress — the actual compression is done by the external compressors you already have.

Includes a GUI wrapper (sycg) with real-time metrics, dark/white themes, custom icons, and InnoSetup integration. Supports solid mode (-tar), multi-part archives, AES-256/ChaCha20 encryption, CRC32/MD5 hashing, and builds for Windows 7–11 x86/x64 using Python 3.8.

Note: This project isn't meant to reinvent the wheel; it's more of a small personal project I created for my own needs that I'd like to share. I CAN'T WAIT TO HEAR YOUR IDEAS !!!

See this: https://encode.su/threads/4480-SYC-M...mpression-Tool

[kj911]

Separately as a new post!

YadeWira released packJPG Multithreaded versions few hours ago! Windows x86/64 and Linux64 binaries in available!

https://encode.su/threads/4482-packJPG-Multi-threaded

[KaktoR]

Great.

I just tested a bit. It looks like -th0 is actually much slower than without this option.

-th0
Compressed 9 files, 6,362,109 => 4,683,122 bytes. Ratio 73.61%
Compression time: cpu 0.00 sec/real 4.06 sec = 0%. Speed 1.57 mB/s
All OK

no -th option
Compressed 9 files, 6,362,109 => 4,683,122 bytes. Ratio 73.61%
Compression time: cpu 0.02 sec/real 2.12 sec = 1%. Speed 2.99 mB/s
All OK

-th6 is the same as -th0. Am I missunderstand this option?

Also you still have to use solid=0 option for freearc sadly.

Edit: I just saw this is the wrong toppic for such discusions.

[YadeWira]

Here’s what I replied on the encode.su forum: “To be honest, I haven’t tried it on FreeArc yet. It’s not recommended to use the -th option with a single file, since it yields worse results in terms of compression and decompression times. In any case, you should create an option that generates a package (.pjgp?) containing all the already-compressed .jpg files or something similar. I don’t know what you think."

[wrathma]

oodlescan

a simple oodle stream scanner i made. 99% of the logic is taken from xtool source code. built with gcc on windows x64. the only reason i made this and difference between this and Razor12911's oo2scan_7_win64 is that this one uses chunk based data loading. oo2scan_7_win64 tries to occupy the entire size of the file on memory to store and scan it from there, in cases where the data is huge it would fail. oodlescan copies small chunks from the file to memory and scans from there, eliminating the memroy allocation issue. i will add the source code here after some polishing .

 SYMM's Oodle Stream Scanner

 - Scans files for Oodle streams

 Usage: H:\test\ooscan\oodlescan.exe [options] <file>

 Options:
   -cN    size of chunks (e.g. -c16M)
   -v     show valid streams only
   -vv    show valid and invalid streams
   -h     show this help message

edit: added source code

[wrathma]

prl

a simple and very lightweight (11kb binary, lzma 5kb) parallel batch processor. built with msvc x64. it lacks features but gets the job done.
source code moved to codeberg

Usage: H:\test\tools\build\prl.exe <input.bat> [thread_count]

edit: update v1
replaced semaphore with a threadpool.
replaced system() calls with CreateProcessA

[wrathma]

wem(un)pak

This is a more feature packed version of Masquerade's WemTool that i made few months ago to compress wem files from Silent Hill f. As it was made with golang, the binary sizes are huge. I didnt want to port this entire thing ito c, so today i added a feature to generate a batch script to decompress the files. Then you can run the batch script in parallel with prl or Mparallel. It follows same file naming standards as original WemTool. So you can use wempack to decompress WemTool compressed data. Attachments include source code.

• Separate compressor and decompressor.
• Ability to compare input and compressed sizes.
• Works recursively.
• Ability to specify compressor for hdiffz.
• Ability to generate a batch script to decompress.
• Better error handling and processing structure:
• compressor : ww2ogg > oggre_enc > oggre_dec > hdiffz
• decompressor : oggre_dec > hdiffz
• If any of the steps fails, the tool skips that file automatically. Temporary files are deleted in every step of processing, WemTool processes all files and then removes temps at the very end, leaving a huge mess and sometimes filling up the disk.

 SYMM's WEM packer

 - Compresses .wem to .ww + .DIFFZ

 Usage: wem-packer.exe [options] <directory>

 Options:
   -tN    number of threads (e.g. -t4/defaults to cpu count)
   -s     skip if output (.ww + .DIFFZ) is larger than input (.wem)
   -b     keep original .wem files
   -cX    compressor for hdiffz, default = none.
   -h     show this help message

 SYMM's WEM unpacker

 - Decompresses .ww + .DIFFZ to .wem

 Usage: wem-unpacker.exe [options] <directory>

 Options:
   -tN    number of threads (e.g. -t4)
   -b     keep original .ww and .DIFFZ files
   -n     generate a batch script to decompress
   -h     show this help message

Credits:

• Masquerade for the guide
• Sisong for hpatchz/hdiffz
• hsc64 for ww2ogg
• Profrager for oggre

[KaktoR]

Originally Posted by wrathma

wem(un)pak

Some small bug

Found 247 .wem files to pack (recursive)
Using 12 threads

[███████████████████████████████████████░] 246/247 (99.6%) ETA: 0s
All files packed successfully!

Also it would be good to give some information on how long the process took.

Another bug in decode mode

Found 389 .ww files to unpack (recursive)
Using 12 threads

[█████████████████████████████████████████████████░] 388/389 (99.7%)
All files unpacked successfully!
[██████████████████████████████████████████████████] 389/389 (100.0%)

Testing:
Surprised how fast this is compared to WemTool

input 339mb
output 316mb

wem-packer
encode 17sec
decode 5sec


wemtool
encode 50sec
decode 16sec

Both tools are running with all cpu threads.

Edit:
Impressive

input 5,12gb
output 4,82gb

wem-packer
encode 10m30s
decode 2m40s


wemtool
encode 30m20s
decode 13min

wempacker is 3x faster than wemtool

Suggestions:
Add -c option for hdiffz compression
Change skip -s option to skip files even if they are same size as input -> equal or greater then

[Masquerade]

Very nice tool wrathma and pleased someone has nulled the need for my bat2exe nightmare creation.

[wrathma]

Originally Posted by KaktoR

Some small bug

Found 247 .wem files to pack (recursive)
Using 12 threads

[███████████████████████████████████████░] 246/247 (99.6%) ETA: 0s
All files packed successfully!

Also it would be good to give some information on how long the process took.

Suggestions:
Add -c option for hdiffz compression
Change skip -s option to skip files even if they are same size as input -> equal or greater then

thanks for your detailed review KaktoR. the progress bar was shamelessly copied from stackoverflow and unfortunately it
doesnt work great with multithreaded operations. so i removed it and added a simple progress bar. should be fixed by now. also
added -c option for hdiffz. everything you pass after this will be passed directly to hdiffz -c-{here}. example -

wem-packer.exe -czstd-22 wemfolder -> hdiffz -c-zstd-22 ...

Originally Posted by kj911

wem-packer: Why are both "oggre_enc/oggre_dec" binaries needed? Also, how are they 9KB larger than the original files?

There is also a "revorb" project, isn't that better than "ww2ogg"? Instead of the HDiff package, would xdelta3 be a worse solution? (it runs natively on XP too.) HDiffPatch XP issues from GitHub. (Two x86_64/i686 XP binaries found the posts.)

Is there a way to configure the program, especially "hdiffz", to use very little memory when creating a diff file? (It should comfortably fit within the 512MB upto 1-2GB memory limit.) It is true that when applying a diff file as a patch, in principle a lot of memory is required.

The only task that remains to be solved is to create an EXE file that can be run natively under XP by the GO-compiler. (Why does this require the "bcryptprimitives.dll" file?)

both oggre encoder and decoder is required because sometimes doing oggre_enc and oggre_dec on a file gives different
files (crc error). but doing oggre_enc on the output of oggre_dec will always give same files. try this out yourself
on diffrent ogg files you will get it.
i have never used revord, i guess ww2ogg works so ill use it for now (rule of engineering, if it works dont touch it).

xdelta3 and hdiffz could be interchanged as they are basically doing the same thing. i have added a xdelta build for
you. keep in mind that xdelta produces a little bigger diff files compared to hdiffz. tho the difference is minor.
during my tests, the entire tool running on 20 threads didnt cross the 500 mb ram usage mark. stayed mostly on
380-400 mb of ram usage (wempak+hdiffz+oggre+ww2ogg). applying patch is easy while making patch is harder and takes
lot of ram. but when the files are small ram usage also goes down.

golang turned down support for windows xp long ago i forgot to mention that so this tool shouldnt really work on
windows xp. i dont even ship wemunpak on my repacks and neither should you. the binary is bloated with golang
bloat code. this is why i added a feature to make a batch file to run the entire decompression chain.
check the -n option. you can then ship the batch file with the tools (hpatchz+oggre) and a
parallel processor (prl or mparallel)

Originally Posted by kj911

wrathma: There are problems with this GOLang miracle! Does it need an AVX/AVX2-compatible processor? Or do the compiled binaries only work flawlessly on the given machine? What if I remove the GO compiler from the machine, will the binaries compiled until then break?

The x64 version also produces similar and longer errors. (Checked on two different Win7 x64 PCs.)
If you chain the compiled program to the given machine/Windows files, it is very bad and makes the given program and the compiler unreliable.
If the source code were rewritten to c/c++, how much better would it be? (With AI, I had a CPP produced, but the program is faulty, it could only serve as some guidance for porting to CPP.)

In its current state, the CPP file only produces "apparently" working programs after compilation!

it doesnt need a avx/avx2 compatible processor to run. atleast the logic should
work on all processors. golang programs are compiled and they are standalone.
unlike python/nodejs you dont need a interpreter to be installed on the user end to run this.

honestly i dont see any reason to rewrite this tool into cpp/c. performance
difference would be minor or nonexistent, but this type of tool is not meant
to be coded in these low level languages. i have seen your cpp code, you mostly
copied my structure a to z but the encoding/decoding logic is not there. if you
complete this on c the binary sizes might be cut to half or even more. and it
would probably work on older systems. if you follow what i do, i pack the files
in a modern device and then ship the tools required with a batch file. well if
you want feel free to rewrite this tool in c/cpp.

[kj911]

wrathma: This wempak tool, XP-compatible or Vista+ 32bit?? Some parts from Masquerade's WemTool, are yes running from XP.

[wrathma]

Originally Posted by kj911

wrathma: This wempak tool, XP-compatible or Vista+ 32bit?? Some parts from Masquerade's WemTool, are yes running from XP.

the one above builds shouldnt work on 32bit systems, i have attached a 32bit builds for you. not sure if it will work correctly as i dont have a 32bit test system laying around. i replaced the 64bit hdiffz/hpatchz exe with 32bit ones, everything else is same.

[kj911]

wrathma: Many questions!

wem-packer: Why are both "oggre_enc/oggre_dec" binaries needed? Also, how are they 9KB larger than the original files?
In the original "ww2ogg" package, there is another "packed_codebooks.bin" file. Could the lack of this cause problems in some cases?
There is a 64-bit version of this program on GitHub, I won't link it directly, find it and take a look, you'll get the hang of it!

There is also a "revorb" project, isn't that better than "ww2ogg"? Instead of the HDiff package, would xdelta3 be a worse solution? (it runs natively on XP too.) HDiffPatch XP issues from GitHub. (Two x86_64/i686 XP binaries found the posts.)

Is there a way to configure the program, especially "hdiffz", to use very little memory when creating a diff file? (It should comfortably fit within the 512MB upto 1-2GB memory limit.) It is true that when applying a diff file as a patch, in principle a lot of memory is required.

You're obviously asking, why XP?? If everything is true, around 2005 (or earlier?) games that have *.wem files were released and regarding these files, it would be possible to make the repacks publishable/archivable back to XP.

There's a bug in the 32bit version! The "hdiffz" binary file in it is 64bit!

Yesterday, I managed to make all the binaries run on XP. Although, a full native XP-compatible HDiff/patchz binaries would be better case/idea.

In the case of v4.8.0, the "hpathcz.exe", with the simple NT6.0 -> NT5.1 trick, runs smoothly, while the larger "hdiffz.exe" file, I had to tinker a bit more with CFF Explorer + 3 extra DLL files required, to run it.

In the case of v4.11+ versions, the latter situation is present for both files.

The only task that remains to be solved is to create an EXE file that can be run natively under XP by the GO-compiler. (Why does this require the "bcryptprimitives.dll" file?)