LZ4 Stream detector (Test project)
 

Hi guys, first of. I'd like to start by saying that lz4 is a shit codec to handle when it comes to universal precompression. 2 years ago, I lost some of my source codes then a few days back, I found them because I had totally forgotten what was really required to make the damn thing work.

I bring this slow af stream detector, try it on a few games.

* NOTE *

You must have experience with game repacking and have knowledge as to how streams/blocks really work.

I say this because this thing can be really slow but depending on what you use it on and how you use it. For example, call of duty has those ff files, their data is stored in 64k blocks, if you open the batch file, you'll find 65536 as default which means it should detect streams from those files very quickly.

Those games that use unity engine normally come with 128k blocks, so you should open bat and give in 131072.

The other thing is, this detector as of right now only detects streams/blocks of the specified size, meaning if a file contains blocks of a different size, those will not be detected, still working on it...

That is all for now so go give it a shot and perhaps upload a few samples if you have any that are interesting that I should look at.

* ROAD MAP *

If all goes well with a couple of tests which I'll be posting, there should be a lz4rec side project that should handle lz4 precompressed games, provided I figured out how to speed it up a bit :)

Thanks for this :)

No lz4hc, right?

doesn't matter, if it's a lz4 stream whether compressed with lz4hc or not it should work

will update game file scaner?

not sure if I still have the source for that thing.

@everyone
17 thanks given... yep keep them coming, I just wonder if half of you even tested this thing :rolleyes:

Well the truth is I don't know what to do with the last 3 numbers :D

I will test a bit around with DOS2EE (the only game I have and know of which use lz4 (even it should be lz4hc)).

sorry, I thought it would be the development of the project.

Great to see you again, I will give it a try in coming days though I have no idea which games that I have use lz4. I will probably look at list of unity games on the net and see if some I have "click".

With that said, I strongly recommend everyone to also try 4096, 8192, 16384 and 32768! It may look a lot but together with 65536 and 131072 it should be all thats needed for 99% cases. Run them all in parallel per file through another custom bat file you make.

Btw Razor, I am pretty sure if this work it can be applied exactly same to zstd, only different library and call names. Although I am not sure how you want to make it universal(game-wise), its not only about version differences but more about structure(crc vs no crc, cmp size, decmp size, positions and ordering of all above etc..).

With that said, universal lz4 rec is certainly missing and yes I agree, its a shitty codec - I personally hate it with passion - both lz4 AND that ugly overrated zstd.