Tools for Repacking

[artag]

i don't know if this is the right place to ask, sorry if my english is bad:

wich is the safest and stable cls for precomp right now? (with stdio support) and which
precomp version should i use?

which is the safest cls for srep? (or maybe should i use srep without it). what version is the most stable until today ? 2.2, 3.93, 4.90?


can i remake my old repacks with xtool and forget about pzlib (for zlib streams of course) or pzlib should be used for old games or something like that?

i used lolz 21a7, is there a new updated version?

inno setup 6xxx is stable with isdone dll?

many thanks!

[Masquerade]

^^
For stdio precomp, you can use this version: https://fileforums.com/showthread.php?t=105410

CLS-SREP by ProFrager is still the best cls-srep for decompression. Use srep.exe for compression. 3.93b is alright for most data.

pzlib is obsolete, use xtool.

lolz v22c4b is the latest lolz version.

Inno Setup 6 does work with isdone, yes.

[artag]

Originally Posted by Masquerade

^^
For stdio precomp, you can use this version: https://fileforums.com/showthread.php?t=105410

CLS-SREP by ProFrager is still the best cls-srep for decompression. Use srep.exe for compression. 3.93b is alright for most data.

pzlib is obsolete, use xtool.

lolz v22c4b is the latest lolz version.

Inno Setup 6 does work with isdone, yes.

sorry, for the srep cls, are you talking about this one?:

https://krinkels.org/resources/cls-srep.261/

or are you talking about srep inside?

i cant donwload from krinkels, can you or anybody provide the md5 of the cls
so i can find it on my folders? (I surely have it, but I must be sure that it is the right one.)

i have lolz v22c4b, is everybody using it? is it stable?

thanks again

[kj911]

wrathma: There are problems with this GOLang miracle! Does it need an AVX/AVX2-compatible processor? Or do the compiled binaries only work flawlessly on the given machine? What if I remove the GO compiler from the machine, will the binaries compiled until then break?

The GO compiler* used. (I also compiled THIS with it, on XP it prints out the Help nicely, that's all, when compiling (go.exe build source.go) it also prints out similar errors as what can be read below)

*Used last v1.26.2-1 i386 packages and separately compiled from sources. (This v1.24.4 from XP) from make EXEcutables.

The x32 binary you posted, writes these (under Win7 SP1 x64):

C:\wemtest>wem-packer.exe
Exception 0xc0000005 0x8 0x0 0x0
PC=0x0

internal/runtime/syscall/windows.asmstdcall(0x20)
        internal/runtime/syscall/windows/asm_windows_386.s:37 +0x2a fp=0x3afa08
sp=0x3afa04 pc=0xd1029a
eax     0x8
ebx     0x117c630
ecx     0x0
edx     0x3afa38
edi     0x3afa0c
esi     0x3afa70
ebp     0x3afa0c
esp     0x3afa00
eip     0x0
eflags  0x10202
cs      0x23
fs      0x53
gs      0x2b

The x64 version also produces similar and longer errors. (Checked on two different Win7 x64 PCs.)
If you chain the compiled program to the given machine/Windows files, it is very bad and makes the given program and the compiler unreliable.
If the source code were rewritten to c/c++, how much better would it be? (With AI, I had a CPP produced, but the program is faulty, it could only serve as some guidance for porting to CPP.)

In its current state, the CPP file only produces "apparently" working programs after compilation!

So far, I have produced 3 different packages from it, but under XP it throws errors similar to the above, the program prints the prompt in the console window regardless. Under Win7 there are no more strange errors, but as it turned out, if you have to wait for IO errors, HDD or the CPU is too busy, then it will make errors, there is a possibility of making errors. On a laptop with a 2-core CPU, not foolproof, on an 8C/16T Xeon PC, it seems to be better. That is, it doesn't unpack the files, it doesn't run the "hpatchz.exe" file at first, or some other anomaly.

First version maked from Win7 via GO v1.26.2-1: Link
Second version maked from Win7 via GO v1.24.4, hopefully targeted WinXP: Link

Masquerade: Do you still have the BAT2EXE program that you used to create the "WemTool.exe" file? Would you create it again? I'm attaching the patched/replaced/updated files. That way it will be 100% compatible with XP!

[panker1992]

i decided to pop some stuff after a long time, not only i updated my archiver to be a beast!

But i run a showcase to demonstrate the vast capabilities it can do!

i decided to run resident evil requiem precompression only

[KaktoR]

Thanks for the update.

Found 1 .wem files to pack (recursive)
Using 12 threads

[0%] 0/1 Files processed, ETA: 0s
All files packed successfully in 0s!

It's just a visual bug, so nothing to deal with on tryhard level.

[kj911]

wrathma: Thanks for the detailed description!

Speaking of switches:

wem-packer: Here, using "-s" or "-s -b" does not cause any particular confusion, during compress, unlike the unpacker.

wem-unpacker: Do not use the "-b" switch here, otherwise it may happen that the first time it is run it will fail (referring to "hdiff/xdelta"), while the second time it will run without errors. It is better to use it without switches, except for the "-tN" switch.

This is related to the fact that the "original" *.wem file is also in the folder, which conflicts with one of the members of the hdiffz+oggre chain and should appear at the end of the process, as the new *.wem file. That is, wem-unpacker deletes the original *.wem file and exits here, without continuing to run the hdiffz+oggre chain to then finish the unpacking properly, as it should.

C:\>wem-packer.exe -s -b music_frontend
Found 4 .wem files to pack (recursive)
Using 2 threads with size check (keeping backups)


268707965.wem: output is bigger than input, skipped

681733649.wem: output is bigger than input, skipped

775589494.wem: output is bigger than input, skipped

All files packed successfully in 7s!
C:\>wem-unpacker.exe music_frontend
Found 1 .ww files to unpack (recursive)
Using 2 threads

[0%] 0/1 Files processed, ETA: 0s
Errors encountered:
  108187332.ww: xdelta failed
Completed with errors in %s
0s
C:\>wem-unpacker.exe music_frontend
Found 1 .ww files to unpack (recursive)
Using 2 threads

[0%] 0/1 Files processed, ETA: 0s
All files unpacked successfully 4s!

wempack-xdelta package: This doesn't run under Win7 x64 either. (Has anyone tested it under Win7, are you using the uploaded files or only Win10/11?)

Running the GO compiler with these switches, I managed to produce somewhat smaller EXEs. It should be 100% compatible with Win7 (also with 32-bit editions, theoretically going back to Vista, up to the early 32-bit version of Win10.) although, all "embedded binaries" are XP-compatible.

go build -a -gcflags=all="-l -B" -ldflags="-w -s" -o myapp main.go

The uploaded CPP file would require a more experienced programmer than me, this was also transpiled with AI, from GO to C++.

UPD: Embedding binary files via alternative methods: https://github.com/samuelngs/binary-go workable from these "wem-(un)packer's" project?

[wrathma]

Originally Posted by kj911

wempack-xdelta package: This doesn't run under Win7 x64 either. (Has anyone tested it under Win7, are you using the uploaded files or only Win10/11?)

that was my mistake. i compiled it with latest golang (v1.26.2) that doesnt
support windows 7. i tried to build it today with golang v1.20 on win10 and
the compiled exe works fine on windows 7. also one thing i noticed, that
xdelta or wemtool also doesnt work on windows xp (i might be missing out
something, check screenshots). i tried to compress the wem files in win 7 x64
vm it worked. then i generated a batch script and sent that to my win xp vm,
and then it gave me some errors that say xdelta wont work. unfortunately
thats something i cant fix.

Originally Posted by kj911

Running the GO compiler with these switches, I managed to produce somewhat smaller EXEs. It should be 100% compatible with Win7 (also with 32-bit editions, theoretically going back to Vista, up to the early 32-bit version of Win10.) although, all "embedded binaries" are XP-compatible.

go build -a -gcflags=all="-l -B" -ldflags="-w -s" -o myapp main.go

UPD: Embedding binary files via alternative methods: https://github.com/samuelngs/binary-go workable from these "wem-(un)packer's" project?

i also used a similar build flag -

go build -ldflags="-s -w" -tags windows -trimpath -gcflags=all="-l -B" ...

i did not use -a because all it does is rebuild external packages (wempack doesnt
use any external packages). and for the alternative binary embedding method, i
dont see any reason touse it. i also used to use this binary-go library long ago but
recent golang updates made this one obsolete. now the functionality is built into
golang. this is what i use (you can see the first few lines of my code) -

//go:embed tools/ww2ogg.exe
var ww2ogg []byte

//go:embed tools/oggre_enc.exe
var oggre_enc []byte

it simply copies the raw data as a a binary stream while compiling the exe. and
when running the tool it simply copies the binary data from the exe to a temp
folder. see extractTools() from line 50-72 of wem-packer.go.

[kj911]

Use the binary files I uploaded! I've sorted them and patched them so they can run under XP! (For use the 32bit wem-(un)packer releases.)

List: WemTool_XP_compat_files.7z (Although this pack was uploaded directly for Masquerade.) You can download the XP compatible "xdelta.exe" from the original site (marked "i686"). Or you can use the files below. (After running, copy them from the TEMP directory.) The my uploaded "mp.exe" XP-compatible.

wempack_x86_hdiff.7z
wempack_x86_xdelta.7z

If all *.exe files, fixed for XP, run without errors, then the batch script should run without errors when compressing and decompressing *.wem files. (I haven't even experimented with this yet.)

Win7 compatible GOLang (I already linked it earlier.): https://github.com/thongtech/go-legacy-win7

[wrathma]

Originally Posted by kj911

If all *.exe files, fixed for XP, run without errors, then the batch script should run without errors when compressing and decompressing *.wem files. (I haven't even experimented with this yet.)

your patched exes works under xp environment. i think you can use
the batch script function of wem-unpacker with your fixed winxp
executables to decompress wem files under winxp environment.

currently im working on editing my prl tool (a mparallel alternative)
to move to win32 system api calls instead of relying on crt libraries.
that way it can run natively on any post winxp system without needing
to install any msvcrt libraries. plus the file size will also shrink (it is still small).

[wrathma]

based on latest github/zstd-1.5.7 sources. needs thorough testing. use 1.5.7 to compress. examples -

> zstd_decompress.exe

 Experimental ZSTD Decompressor by SYMM

 usage: zstd_decompressor input output
 replace input and/or output with - to use stdin/stdout

> zstd_decompress.exe input.zst output.exe
output.exe : 126607283 bytes

zstd_decompress.exe - - < input.zst > output.exe
<stdin>: 126607283 bytes written

on benchmarks its almost 2x slower than the official zstd binary release (1.5mb). still it is fast with almost 350MBps decompression speed for 4GB uncompressed data on my test system (4c8t i3) on nvme.

[newfolder]

Unity Precomp, the open source tool to precompress unity3d/bundle files, includes the binary and the source code in c

[newfolder]

Unity Decomp is a program that decompresses Unity bundle files, essentially expanding them so the expanded file is compatible with the Unity engine. This is ideal for people who want to optimize compression, since removing the lz4/lz4hc data allows you to use better compressors like 7-Zip, LOLZ, or FreeARC.

Includes the binary and the source code in c

[wrathma]

this is a standalone bz3 decompressor. based on latest iczelia/bzip3
commit 97a6da2. use bzip3 v1.5.3 to compress. decompression
speed same as official bzip3 release.

tried to build with winxp support but failed. i dont think bzip3 source
codes support winxp at all. eventually got the standalone compiled
x86 exe down to 35kb (no ucrt/msvcrt dll dependency) but it wouldnt
work on win7 without ucrt. worked fine on 10+ tho. so had to add
ucrt. works on all win7+ systems even without any msvcrt installation.

attachment includes source code. to compile make a decompress/ dir
inside bzip3/ git repo and copy the attached CMakeLists.txt and
main_decompress.c and then build the project using cmake.

> bunzip3_x86.exe

 bz3 decompressor by SYMM

 usage: bunzip3 input output
 replace input and/or output with - to use stdin/stdout

> bunzip3_x86.exe input.bz3 output.exe
output.exe : 824130 bytes

> bunzip3_x86.exe - - < input.bz3 > output.exe
<stdin>: 824130 bytes

[kj911]

BUNZIP3: A question arose, how can it not get an error if the "libsais" library is not included?

Please note that if you are unpacking these *.bz3 archives on a 32-bit machine, do not use a large block size and/or multiple threads, otherwise it will crash and/or exit silently, resulting in a 0-byte file.

Packed in "-b 511" switches:
x64 exe only one capable it! x86 version (include my XP-versions) still crashing!

Packed in 320MB's block size:
x64 exe: works.
x86: silently exit and will get 0 byte file.
x86_laa: only one capable ~2 million KB memory allocation from x64 Windows host, perfectly decompress.

Packed in 255MB's block size:
all x86 exe hopefully works, included all x32 systems.

Recommendedly use max. 255MB's block size or failsafely smaller values from targeting real 32bit systems. (ex: 64/128/192/224MB.)
These "-b 255" packed archives, I need ~1 600 000KB free memory allocation.
Tested from "enwik9" file.