wrathma: Thanks for the detailed description!

Speaking of switches:

wem-packer: Here, using "-s" or "-s -b" does not cause any particular confusion, during compress, unlike the unpacker.

wem-unpacker: Do not use the "-b" switch here, otherwise it may happen that the first time it is run it will fail (referring to "hdiff/xdelta"), while the second time it will run without errors. It is better to use it without switches, except for the "-tN" switch.

This is related to the fact that the "original" *.wem file is also in the folder, which conflicts with one of the members of the hdiffz+oggre chain and should appear at the end of the process, as the new *.wem file. That is, wem-unpacker deletes the original *.wem file and exits here, without continuing to run the hdiffz+oggre chain to then finish the unpacking properly, as it should.

wempack-xdelta package: This doesn't run under Win7 x64 either. (Has anyone tested it under Win7, are you using the uploaded files or only Win10/11?)

Running the GO compiler with these switches, I managed to produce somewhat smaller EXEs. It should be 100% compatible with Win7 (also with 32-bit editions, theoretically going back to Vista, up to the early 32-bit version of Win10.) although, all "embedded binaries" are XP-compatible.
The uploaded CPP file would require a more experienced programmer than me, this was also transpiled with AI, from GO to C++.

UPD: Embedding binary files via alternative methods: https://github.com/samuelngs/binary-go workable from these "wem-(un)packer's" project?

that was my mistake. i compiled it with latest golang (v1.26.2) that doesnt
support windows 7. i tried to build it today with golang v1.20 on win10 and
the compiled exe works fine on windows 7. also one thing i noticed, that
xdelta or wemtool also doesnt work on windows xp (i might be missing out
something, check screenshots). i tried to compress the wem files in win 7 x64
vm it worked. then i generated a batch script and sent that to my win xp vm,
and then it gave me some errors that say xdelta wont work. unfortunately
thats something i cant fix.

i also used a similar build flag -
i did not use -a because all it does is rebuild external packages (wempack doesnt
use any external packages). and for the alternative binary embedding method, i
dont see any reason touse it. i also used to use this binary-go library long ago but
recent golang updates made this one obsolete. now the functionality is built into
golang. this is what i use (you can see the first few lines of my code) -

it simply copies the raw data as a a binary stream while compiling the exe. and
when running the tool it simply copies the binary data from the exe to a temp
folder. see extractTools() from line 50-72 of wem-packer.go.

https://i.ibb.co/JwmrJvTM/Virtual-Bo...6-16-50-07.png
https://i.ibb.co/YBwM5YWM/Virtual-Bo...6-23-48-37.png
https://i.ibb.co/N2m1sqwL/Virtual-Bo...6-00-00-15.png

Use the binary files I uploaded! I've sorted them and patched them so they can run under XP! (For use the 32bit wem-(un)packer releases.)

List: WemTool_XP_compat_files.7z (Although this pack was uploaded directly for Masquerade.) You can download the XP compatible "xdelta.exe" from the original site (marked "i686"). Or you can use the files below. (After running, copy them from the TEMP directory.) The my uploaded "mp.exe" XP-compatible.

wempack_x86_hdiff.7z
wempack_x86_xdelta.7z

If all *.exe files, fixed for XP, run without errors, then the batch script should run without errors when compressing and decompressing *.wem files. (I haven't even experimented with this yet.)

Win7 compatible GOLang (I already linked it earlier.): https://github.com/thongtech/go-legacy-win7

your patched exes works under xp environment. i think you can use
the batch script function of wem-unpacker with your fixed winxp
executables to decompress wem files under winxp environment.

currently im working on editing my prl tool (a mparallel alternative)
to move to win32 system api calls instead of relying on crt libraries.
that way it can run natively on any post winxp system without needing
to install any msvcrt libraries. plus the file size will also shrink (it is still small).

Experimental ZSTD decompressor
 

based on latest github/zstd-1.5.7 sources. needs thorough testing. use 1.5.7 to compress. examples -
on benchmarks its almost 2x slower than the official zstd binary release (1.5mb). still it is fast with almost 350MBps decompression speed for 4GB uncompressed data on my test system (4c8t i3) on nvme.

Unity Precomp, the open source tool to precompress unity3d/bundle files, includes the binary and the source code in c

Unity Decomp is a program that decompresses Unity bundle files, essentially expanding them so the expanded file is compatible with the Unity engine. This is ideal for people who want to optimize compression, since removing the lz4/lz4hc data allows you to use better compressors like 7-Zip, LOLZ, or FreeARC.

Includes the binary and the source code in c

bunzip3
 

this is a standalone bz3 decompressor. based on latest iczelia/bzip3
commit 97a6da2. use bzip3 v1.5.3 to compress. decompression
speed same as official bzip3 release.

tried to build with winxp support but failed. i dont think bzip3 source
codes support winxp at all. eventually got the standalone compiled
x86 exe down to 35kb (no ucrt/msvcrt dll dependency) but it wouldnt
work on win7 without ucrt. worked fine on 10+ tho. so had to add
ucrt. works on all win7+ systems even without any msvcrt installation.

attachment includes source code. to compile make a decompress/ dir
inside bzip3/ git repo and copy the attached CMakeLists.txt and
main_decompress.c and then build the project using cmake.

BUNZIP3: A question arose, how can it not get an error if the "libsais" library is not included?

Please note that if you are unpacking these *.bz3 archives on a 32-bit machine, do not use a large block size and/or multiple threads, otherwise it will crash and/or exit silently, resulting in a 0-byte file.

Packed in "-b 511" switches:
x64 exe only one capable it! x86 version (include my XP-versions) still crashing!

Packed in 320MB's block size:
x64 exe: works.
x86: silently exit and will get 0 byte file.
x86_laa: only one capable ~2 million KB memory allocation from x64 Windows host, perfectly decompress.

Packed in 255MB's block size:
all x86 exe hopefully works, included all x32 systems.

Recommendedly use max. 255MB's block size or failsafely smaller values from targeting real 32bit systems. (ex: 64/128/192/224MB.)
These "-b 255" packed archives, I need ~1 600 000KB free memory allocation.
Tested from "enwik9" file.

if you see at the top of CMakeLists.txt youll find that it includes src/libbz3.c in the build command which imports libsais.h :)
btw i tried your winxp builds and it works fine on my vm, can you share how you did that ? did you use very old visual studio ?

The source code wouldn't compile by hand... (Neither with MinGW nor with w64devkit.)

The main solution was (if all files are in the correct right place), that I had to generate a MinGW-compatible "Makefile" from the "CMakeLists.txt" file using "cmake" in "w64devkit". (last v2.6.0 or 2.7.0 with GCC15.2.0. I'm looking now, v2.8.0 has already been released.)

In principle it was this: cmake -G "MinGW Makefiles" "CMakeLists.txt"

After that, I just had to run "mingw32-make.exe" and the binary file was ready. (After that, I cut out the excess with "strip.exe".)
"*_laa.exe" was created by patching the original file with a small program called "4gb_patch".

w64devkit can also build XP-compatible EXEs. (With some luck.)

These lines changed to:
/SUBSYSTEM:CONSOLE,6.01 --> 5.01
not resolve and generate XP-compatible binaries?

Visual Studio?? As far as I know, it's very difficult to get this nowadays (let alone install it offline), not to mention the many GB of storage space required...

By any chance, you don't have the xp v141 package on your machine? There might be a couple of CAB compression sources. What does it actually do, compared to makecab? Rust, you don't have it installed by any chance?

i have v141_xp msvc toolset. i tried atleast 20 times with different compiler/linker
args to try to add xp support. yes i also tried /SUBSYSTEM:CONSOLE,5.01 it doesnt
help much. as i have seen using it and not using it is the same. if it were to run on
winxp it would run without this. btw even if you have v141_xp toolset, cmake will
try to target latest toolset. you have to specify it before building like this.
yeah it is heavy. visual studio is taking up almost 50gb on my system drive.
but thanks for the tip, i never knew using non msvc compiler could fix it.

CMake can, in principle, be configured to always use the given "toolkit" to compile the given source code. (ex: Ninja, MinGW, VisualStudio, etc...)

To be honest, I don't have much experience with these programs,.. If, the package is put together well, the desired/tested program can usually be produced, if there is source code for it.

Manual patching of EXEs does not always solve the problem of whether they can be run on XP? (If, Vista+ compatible EXEs are released after compilation.) If there are no significant number of Vista/7 kernel-specific calls in the binary file, then they usually work on XP as well.

RAZOR "true" stdio patch
 

so this patched dll uses true stdio (no fileio) compression. directly
from ram to ram. it doesnt write anything to disk by itself.
as it is ram to ram, it will use more memory compared to fileio.

maximum memory usage during compression - input filesize +
output file size + compression memory. maximum memory
usage during decompression - decompression memory + ~4mb.

notes:

1. as i hooked multiple fileio functions of win32api, normal fileio
   compression/decompression through razor is not possible (with
   my patch). using fileio with my patch may lead to bsod. so i have
   added some checks to prevent accidental fileio operations. to do
   fileio operations use the original rz.exe.
2. you might see my patch reading/writing V_IN.SYMM or
   V_OUT.SYMM files, those are just my handles for virtual memory
   address. not actual files on disk.
3. its NOT interchangeable with Razor12911's patch or my old
   "fake" stdio patch.

compiled with mingw-clang. will add the source code after some
cleaning. works fine with arc. added sample arc.ini. for usage
instructions see test.bat/arc.ini.

edit:
fixed a memory leak issue (idk how it slipped past me). removed
debug-specific checks that are not required now. some cosmetic
changes and removed unused debug code. also fixed a bug where
it would fail with freearc decompression.
during testing i found out that razor decompressor is sequential so
made this patch do sequential streamed decompression. so memory
usage is same as (+ ~4 mb) non stdio operations. optimized dll
size (115kb to 14 kb).

edit2:
fixed a small bug, decode speed increased by ~1%. and even more optimized dll size.

If you don't mind, what's the difference between this and https://www.fileforums.com/showpost....81&postcount=8 ? I use razor all the time that's why just asking if i should use this from now on :)

The difference is no temp file copy which simply means it is faster. Not compression itself but it simply doesn't make any temp files. If you use this to compress files on HDD it should make a clear difference in overall compression times. With the old stdio patch rz.exe will copy temp from HDD to HDD (same as lolz does). This step is obsolete. There is nothing faster than RAM copy.

The other day I noticed that the original Razor compressor (rz.exe) created a 32MB temporary file in the %TEMP% folder while testing a 7+ GB archive. Maybe because it would have run faster, testing/unpacking the compressed data, than the HDD speed.

Could someone port Razor to WinXP?? There is an early, v0-1 "draft" source code, heaven knows where it came from. The point is that the program can be compiled for XP, but it doesn't matter what kind of compiler. The MinGW version compresses EXEs worse than the "draft" EXE. This one runs on XP with a little patching. It only handles one file, it's not "fuzzsafe" or whatever they call it. The compression difference is 3-4% worse in favor of the v1.0x series, with a dictionary size of 64MB. There is also a decoder/decompressor compatible with the original RZ, but whether this can be ported to WinXP is a good question.

^^
Maybe you'd get a better answer asking on encode's forum.

if anyone is using my patch i will highly suggest to update it
as some bugs and dead code unfortunately slipped past me
on my last build. a bsod-level issue was fixed. and overall
dll size dropped by almost 90%. will try to be more careful
from now.

as KaktoR said, all it does is compress from ram to ram
eliminating the need to copy stdin to a temp file. if you
try using it on a hdd you will understand the difference.
decompression is same as Razor12911's build.
also a smaller dll wouldnt hurt i think (1.6mb vs 14kb) :D

out of my skillset :(

OodleForge v33.1 - OodleLZ Exact-Match Precompressor (open source)
 

OodleForge v33.3 Engine-Specific Compatibility, Expanded Method Support & Scanner Robustness - OodleLZ Exact-Match Precompressor (open source)

instead of -ssd -hddslow -hddfast you could add a in memory buffer. a set size of memory will be allocated at start (ex: 64mb). you first write processed output to this memory buffer and when its full you flush it directly to storage (1 write operation, works on a separate thread) and in the meantime allocate another memory buffer to start again. so instead of a 1gb file doing 4k+ write operations (with -hdd*) it will do only 16-17 write operations.

You right, this changed the code 100%, amazing

It's good to see an alternative project in the works and given that it is open source, I would assume it is active development? If so, then I'd like to report a few issues I've discovered from the few tests I've conducted.

Issues

- A significant number of streams are left behind (not detected), scanner needs some improvements
- Even though there's only one game known to use of internal crc checks sendQuantumCRCs (The Crew 2), such data cannot be processed by this project
- spaceSpeedTradeoffBytes cannot be set by user and/or program does not check input for this change meaning games that this program cannot work on games that use the frostbite engine
- this project suffers from the same problem as xtool with leviathan and some hydra streams where they can be detected but not processed at all


The streams that are left behind can be however seen by this tool which can be found here, it's good that it is also open source as well, should help with development. :)

Thanks a lot for checking it out and for the detailed feedback — really appreciate it coming from the XTool author!
The public repo is only ~2 days old (still v33.1), but I'm actively developing the Trinity Engine branch (Oodle + Unity LZ4 + Zlib + Preflate) with perfect AES roundtrips.
Scanner improvements are my top priority right now. Would love to hear more details about the missed streams or any test files you can share.
Open to collaboration!

https://github.com/johna124/Oodleforge

Could someone make a 64-bit version of this "t5lzma.exe" (source code) program? (While maintaining compatibility.) The 32-bit version only supports dictionary sizes up to "-d27", it no longer supports "-d28", although it is a good question whether it could handle higher ones. It is noticeably slower than the classic 7z/LZMA, but perhaps it could be minimally better.

An interesting precomp-fork, 32bit test package. (Only suitable for zlib-streams.)

Its compression speed, thread-dependent, can be up to 50+ MB/s. The decompression speed is slightly minimally faster. (Unfortunately, it also has quite strong limitations.)

Original author's description:
See my "readme2.txt" file from few infos. Sourced from this site posts.

Archiver, Demo,

Archiver has in-built a patch engine which allows you to create a patch from 2 folders, and then make it into a proprietary format,

The format itself is designed to be streaming, compressed and deduplicated, precompression failed for this as the patches themselves fail to give proper data which precompression can understand.

the reason is because the patch itself isnt aligned data which each algo is designed to work with.


What this streaming, delivers is a seamless performance which will decompress, edit each file verify it and delete the old one, you can use the folder to patch as well in case the you dont like the idea of having a prebaked archive format,

The API itself in Inno has the ability to take in such a format and applying it, and still maintaining the progress in a progressbar.

Added Edit:
Archiver IO Massive, exec2 in the works


@echo off

archiver i -exec2 -c0="D:\Epica" -c1="ffmpeg.exe" -c2="-i" -c4="-vn -b:a 256k -y" -c5="mp3" -c6=".ogg"

pause