Tools for Repacking

[KaktoR]

Originally Posted by wrathma

wem(un)pak

Some small bug

Found 247 .wem files to pack (recursive)
Using 12 threads

[███████████████████████████████████████░] 246/247 (99.6%) ETA: 0s
All files packed successfully!

Also it would be good to give some information on how long the process took.

Another bug in decode mode

Found 389 .ww files to unpack (recursive)
Using 12 threads

[█████████████████████████████████████████████████░] 388/389 (99.7%)
All files unpacked successfully!
[██████████████████████████████████████████████████] 389/389 (100.0%)

Testing:
Surprised how fast this is compared to WemTool

input 339mb
output 316mb

wem-packer
encode 17sec
decode 5sec


wemtool
encode 50sec
decode 16sec

Both tools are running with all cpu threads.

Edit:
Impressive

input 5,12gb
output 4,82gb

wem-packer
encode 10m30s
decode 2m40s


wemtool
encode 30m20s
decode 13min

wempacker is 3x faster than wemtool

Suggestions:
Add -c option for hdiffz compression
Change skip -s option to skip files even if they are same size as input -> equal or greater then

[Masquerade]

Very nice tool wrathma and pleased someone has nulled the need for my bat2exe nightmare creation.

[wrathma]

Originally Posted by KaktoR

Some small bug

Found 247 .wem files to pack (recursive)
Using 12 threads

[███████████████████████████████████████░] 246/247 (99.6%) ETA: 0s
All files packed successfully!

Also it would be good to give some information on how long the process took.

Suggestions:
Add -c option for hdiffz compression
Change skip -s option to skip files even if they are same size as input -> equal or greater then

thanks for your detailed review KaktoR. the progress bar was shamelessly copied from stackoverflow and unfortunately it
doesnt work great with multithreaded operations. so i removed it and added a simple progress bar. should be fixed by now. also
added -c option for hdiffz. everything you pass after this will be passed directly to hdiffz -c-{here}. example -

wem-packer.exe -czstd-22 wemfolder -> hdiffz -c-zstd-22 ...

Originally Posted by kj911

wem-packer: Why are both "oggre_enc/oggre_dec" binaries needed? Also, how are they 9KB larger than the original files?

There is also a "revorb" project, isn't that better than "ww2ogg"? Instead of the HDiff package, would xdelta3 be a worse solution? (it runs natively on XP too.) HDiffPatch XP issues from GitHub. (Two x86_64/i686 XP binaries found the posts.)

Is there a way to configure the program, especially "hdiffz", to use very little memory when creating a diff file? (It should comfortably fit within the 512MB upto 1-2GB memory limit.) It is true that when applying a diff file as a patch, in principle a lot of memory is required.

The only task that remains to be solved is to create an EXE file that can be run natively under XP by the GO-compiler. (Why does this require the "bcryptprimitives.dll" file?)

both oggre encoder and decoder is required because sometimes doing oggre_enc and oggre_dec on a file gives different
files (crc error). but doing oggre_enc on the output of oggre_dec will always give same files. try this out yourself
on diffrent ogg files you will get it.
i have never used revord, i guess ww2ogg works so ill use it for now (rule of engineering, if it works dont touch it).

xdelta3 and hdiffz could be interchanged as they are basically doing the same thing. i have added a xdelta build for
you. keep in mind that xdelta produces a little bigger diff files compared to hdiffz. tho the difference is minor.
during my tests, the entire tool running on 20 threads didnt cross the 500 mb ram usage mark. stayed mostly on
380-400 mb of ram usage (wempak+hdiffz+oggre+ww2ogg). applying patch is easy while making patch is harder and takes
lot of ram. but when the files are small ram usage also goes down.

golang turned down support for windows xp long ago i forgot to mention that so this tool shouldnt really work on
windows xp. i dont even ship wemunpak on my repacks and neither should you. the binary is bloated with golang
bloat code. this is why i added a feature to make a batch file to run the entire decompression chain.
check the -n option. you can then ship the batch file with the tools (hpatchz+oggre) and a
parallel processor (prl or mparallel)

Originally Posted by kj911

wrathma: There are problems with this GOLang miracle! Does it need an AVX/AVX2-compatible processor? Or do the compiled binaries only work flawlessly on the given machine? What if I remove the GO compiler from the machine, will the binaries compiled until then break?

The x64 version also produces similar and longer errors. (Checked on two different Win7 x64 PCs.)
If you chain the compiled program to the given machine/Windows files, it is very bad and makes the given program and the compiler unreliable.
If the source code were rewritten to c/c++, how much better would it be? (With AI, I had a CPP produced, but the program is faulty, it could only serve as some guidance for porting to CPP.)

In its current state, the CPP file only produces "apparently" working programs after compilation!

it doesnt need a avx/avx2 compatible processor to run. atleast the logic should
work on all processors. golang programs are compiled and they are standalone.
unlike python/nodejs you dont need a interpreter to be installed on the user end to run this.

honestly i dont see any reason to rewrite this tool into cpp/c. performance
difference would be minor or nonexistent, but this type of tool is not meant
to be coded in these low level languages. i have seen your cpp code, you mostly
copied my structure a to z but the encoding/decoding logic is not there. if you
complete this on c the binary sizes might be cut to half or even more. and it
would probably work on older systems. if you follow what i do, i pack the files
in a modern device and then ship the tools required with a batch file. well if
you want feel free to rewrite this tool in c/cpp.