Oodle precompressor (Side project)
 

Hello, I decided to give oo2core_6_win64.dll another shot to see what was really causing the crashing issues I was facing and somehow it the coding software itself so I decided to rewrite the code.... again... but now using Lazarus instead of Delphi because of it causing too many problems, I spent an entire day writing this because it was the only time I had, been very busy lately and I think I got it working.

I have tested on the samples I got from WWE2K19, it handles it just fine, not sure about the entire game but that's for you to find out.

If you look at the attachment, there is a separate jc4 executable. Well that's just a recompile of the main executable just with a few changes to it supports the remaining stream that game has since two compression methods were used on it but to make it work, you would have to combine the main precompressor with the jc4 to fully precompress the game.... I think... not sure since I really had very little time to work on this plus only had 2 samples from the game.
Combining the methods means -moo2rec+oo2recj

Support for now is simply limited to kraken compressed games, just take the dll from game and place it near exe, use the scanner included as a guide, it will show you exactly what method was used in that particular game.

DLLs supported:
oo2core_4_win64.dll
oo2core_5_win64.dll
oo2core_6_win64.dll
oo2core_7_win64.dll

Some results:

Just Cause 4 (game0.arc)
Just Cause 4 (dlc.arc)
These results are already making me wonder if I made mistakes somewhere and left several streams behind.... XDD

See you guys in a while, enjoy and I hope it works lol :)

I would say, better than nothing :)

Thanks for this. Will test soon.

See ya

JC4 Test
 

Thanks Razor12911

And also does not work with the file set
http://uupload.ir/files/ks4s_0.png
http://uupload.ir/files/w2bw_00.png

Thanks Razor12911

I tried the file and folder and extension that address

Just Cause 4\archives_win64\main/game0.arc

oo2core_7_win64.dll Rename oo2core_4_win64.d

game0.arc/oo2rec:
-----------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 890,961,920 => 2,566,282,309 bytes. Ratio 288.04%
Compression time: cpu 0.75 sec/real 51.28 sec = 1%. Speed 17.37 mB/s
All OK

game0.arc/oo2recj:
-----------------------------------------------------------------------------------
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 890,961,920 => 890,962,140 bytes. Ratio 100.00%
Compression time: cpu 0.77 sec/real 6.07 sec = 13%. Speed 146.81 mB/s
All OK

Seems wrong to me, you aren't supposed to use oo2recj individually/separately on a file. You have to use oo2recj in combination with oo2rec. For example = arc a -moo2rec+oo2recj game0.test game0.arc

Update available

Changes

- Fixed crashing issues
- Improved scanner
- Added mermaid precompressor, tested on AC Origins, try it out on Odyssey (if it uses oodle in the first place) and report back

Info

oo2reck = kraken precompressor
oo2recm = mermaid precompressor
oo2recs = selkie precompressor
// reason for them being separated is, there is no stats handler being added in the source yet meaning if I combined all, precompression speed will be reduced drastically so they are left individually at the moment
// if you're wondering what happened to jc4, just use selkie instead, its the same thing. so you should have oo2reck+oo2recs for that game

Future changes

- Detect proper uncompressed sizes resulting in all streams being detected because currently, a few streams get left behind
- Combine all codecs

Just Cause 4 some .it doesn't work with ARC files
is there any solution ?

FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 890,974,208 => 890,974,648 bytes. Ratio 100.00%
Compression time: cpu 0.89 sec/real 16.25 sec = 5%. Speed 54.82 mB/s
All OK

what does the scanner report on that file?

http://uupload.ir/files/ks4s_0.png

Universal Oodle stream scanner

Created by Razor12911

[0] = Unknown/Invalid
[1] = Kraken/Hydra
[2] = Mermaid/Selkie/Hydra
[3] = Leviathan/Hydra

0001[0]| Pos: 00152FAD, Size: 72
0002[0]| Pos: 032C3880, Size: 245
0003[0]| Pos: 03701B74, Size: 2770
0004[0]| Pos: 038C8B5C, Size: 619
0005[0]| Pos: 043CA9F1, Size: 124
0006[0]| Pos: 046AC88B, Size: 3337
0007[0]| Pos: 06B91B94, Size: 3309
0008[0]| Pos: 07A9EC0F, Size: 997
0009[0]| Pos: 08014A4B, Size: 1845
0010[0]| Pos: 0885C8CE, Size: 4019
0011[0]| Pos: 096BBDD4, Size: 2393
0012[0]| Pos: 0C109512, Size: 1038
0013[0]| Pos: 0D585CA3, Size: 2258
0014[0]| Pos: 0EC8F2DB, Size: 1938
0015[0]| Pos: 0FFB679F, Size: 967
0016[0]| Pos: 107CA519, Size: 2362
0017[0]| Pos: 111B2EC9, Size: 709
0018[0]| Pos: 117B5F73, Size: 1423
0019[0]| Pos: 12C24BB4, Size: 2389
0020[0]| Pos: 149D742B, Size: 306
0021[0]| Pos: 14AEBC77, Size: 2512
0022[0]| Pos: 152FE676, Size: 1551
0023[0]| Pos: 160E4C0C, Size: 3829
0024[0]| Pos: 1629FCA3, Size: 1395
0025[0]| Pos: 166F24E7, Size: 840
0026[0]| Pos: 183960EC, Size: 70
0027[0]| Pos: 18B88BCA, Size: 3719
0028[0]| Pos: 198B1CFC, Size: 2212
0029[0]| Pos: 199B27A1, Size: 1864
0030[0]| Pos: 1A1EBCE1, Size: 3107
0031[0]| Pos: 1A3450BE, Size: 2669
0032[0]| Pos: 1C477707, Size: 1787
0033[0]| Pos: 1C729F37, Size: 453
0034[0]| Pos: 1DD43169, Size: 1777
0035[0]| Pos: 1E007F0E, Size: 124
0036[0]| Pos: 1E7B17C6, Size: 1189
0037[0]| Pos: 1EADBDAA, Size: 2176
0038[0]| Pos: 1F4DFBEA, Size: 3518
0039[0]| Pos: 1FB63B0A, Size: 3883
0040[0]| Pos: 1FB97B22, Size: 918
0041[0]| Pos: 1FD6DE7F, Size: 147
0042[0]| Pos: 2008869C, Size: 1099
0043[0]| Pos: 207CAEBD, Size: 613
0044[0]| Pos: 208603A4, Size: 745
0045[0]| Pos: 21954CE5, Size: 730
0046[0]| Pos: 21C40094, Size: 2759
0047[0]| Pos: 231C037C, Size: 1274
0048[0]| Pos: 23CC222A, Size: 273
0049[0]| Pos: 25C3DA65, Size: 1583
0050[0]| Pos: 26720D71, Size: 3208
0051[0]| Pos: 2796EF93, Size: 2358
0052[0]| Pos: 28C09C22, Size: 3577
0053[0]| Pos: 2901B338, Size: 2377
0054[0]| Pos: 2937140C, Size: 554
0055[0]| Pos: 29D6D202, Size: 3658
0056[0]| Pos: 2ACEEB8D, Size: 2176
0057[0]| Pos: 2B0035E5, Size: 473
0058[0]| Pos: 2B9FC064, Size: 2274
0059[0]| Pos: 2C3C3A96, Size: 760
0060[0]| Pos: 2C4DC365, Size: 837
0061[0]| Pos: 2C652A4C, Size: 1418
0062[0]| Pos: 2CA91096, Size: 1234
0063[0]| Pos: 2E36FCA2, Size: 1134
0064[0]| Pos: 2EC756A5, Size: 3169
0065[0]| Pos: 2FC01316, Size: 2357
0066[0]| Pos: 305D0FBA, Size: 1812
0067[0]| Pos: 322B7348, Size: 2980
0068[0]| Pos: 325FDFB2, Size: 3798
0069[0]| Pos: 32869832, Size: 2721
0070[0]| Pos: 3309B82C, Size: 2073
0071[0]| Pos: 3325B12C, Size: 73

Done!!!

Probably no streams are on that file or the scanner doesn't know how to find them.

hmm, so mermaid exists too, mind officially blown.
I guess blend all 3 codecs then for that game. I give up really.
The samples I received only had two codecs used on them.

C:\Users\RAMAZAN\Desktop\Yeni klasör (2)>arc.exe a -ep1 -r -ed -s; -w.\temp -moo2recm data.arc "pack\*"
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 890,974,208 => 890,974,428 bytes. Ratio 100.00%
Compression time: cpu 0.72 sec/real 14.98 sec = 5%. Speed 59.46 mB/s
All OK

https://i.hizliresim.com/0R70QW.jpg

Just Cause 4\archives_win64\main_patch/game4.arc

C:\Users\RAMAZAN\Desktop\Yeni klasör (2)>arc.exe a -ep1 -r -ed -s; -w.\temp -moo2recm data.arc "pack\*"
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 890,974,208 => 890,974,428 bytes. Ratio 100.00%
Compression time: cpu 0.86 sec/real 18.27 sec = 5%. Speed 48.76 mB/s
All OK

Sometimes i get errors like this:

https://i.postimg.cc/MKbp6Brg/Immagine.png

Where is the problem ? Translating "Impossible access to the file. The file is used from another process."

For Just Cause 4, to remove confusion/crashes make sure you only have oo2core_7_win64.dll (remove other/older oo2core dlls) present alongside oo2reck,oo2recm,oo2recs. Now keep in mind that Just Cause 4 mostly uses Kraken so most precompression will come from oo2reck then whatever's left behind will be catched up by Selkie (oo2recs). Mermaid is a stronger oodle codec in terms of compression and there are rarely (perhaps none) any streams of it in the game except for hi-res textures (hires folder) where stronger compression is required.

So my suggestion is use: oo2reck+oo2recs on game files and use oo2recm on hires folder.

Result: oo2reck+oo2recs+srep+lolz = 17.2 GB without hi-res textures, credits (first bink video stream inside game37.arc), useless videos (2.66 GB, embedded inside game files). Best thing ? installation goes really fast, finishes in 30-35 mins on 4 cores+hdd if I remember correctly.

Now, inside "main" folder = game37,38,39,40,41.arc files only contains bink video files so precompression is useless on them. Also inside "main_patch" folder = game6,7,8,9,10.arc only contains bink video files so precompression is useless on them as well. So you may wanna separate them in a different folder.

Read above reply, don't use oo2recm on game files except for hires folder.

Again why do you keep using oo2recm ?

Why rename :confused: Just Cause 4 only has oo2core_7_win64.dll which is whats required by razor's tool. I would recommend not to put any other dlls for this game.

Make sure only oo2core_7_win64.dll (remove other oo2core dlls) is present in your compression folder when compressing Just Cause 4.

I'm using the exact same tools downloaded from Razor main post and here "oo2recm" can't precompress those files, it doesn't find any streams. So the final size is the same of initial size. So how is possible that you precompressed them to "7,249,249,802 bytes" ?

Wolfenstein II The New Colossus Test
 

gameresources.texdb 7.49 GB > oo2core_5_win64 > 32.1 GB:eek:

Creating archive: DataTest.arc using oo2reck+oo2recs:
Compressed 1 Files, 8,055,022,894 => 34,557,525,188 bytes. Ratio 429.02%
Compression time: cpu 69.16 sec/real 4434.67 sec = 2%. Speed 1.82 mB/s
All OK

Testing archive: DataTest.arc
Tested 1 Files, 34,557,525,188 => 8,055,022,894 bytes. Ratio 429.02%
Testing time: cpu 41.08 sec/real 1496.40 sec = 3%. Speed 5.38 mB/s
All OK
http://uupload.ir/files/keh9_0.png

Simorq
What OS are you using and PC specs?

6 Cores 3.2GHz (RZ 1600) + 16GB Ram + HDD
Win 10

How about you L0v3craft?
OS and specs?

Windows Server 2016, intel i7-8700, 64GB ram, 2 SSD 500GB NVME

XCom2 War Of The Chosen Test
 

CharTextures_XPACK_.tfc 1.68 GB >oo2core4 > 3.01 GB


Creating archive: Data.arc using oo2reck
Compressed 1 file, 1,811,671,405 => 3,275,784,960 bytes. Ratio 180.82%
Compression time: cpu 1.77 sec/real 88.45 sec = 2%. Speed 20.48 mB/s
All OK

Testing archive: Data.arc
Testing time: cpu 3.42 sec/real 55.96 sec = 6%. Speed 32.37 mB/s
All OK
======================================

Creating archive: Data.arc using oo2recm
Compressed 1 file, 1,811,671,405 => 1,811,704,170 bytes. Ratio 100.00%
Compression time: cpu 1.73 sec/real 37.30 sec = 5%. Speed 48.57 mB/s
All OK

======================================

Creating archive: Data.arc using oo2recs
Compressed 1 file, 1,811,671,405 => 1,811,671,841 bytes. Ratio 100.00%
Compression time: cpu 1.70 sec/real 36.26 sec = 5%. Speed 49.96 mB/s
All OK

======================================

Creating archive: Data.arc using oo2reck+oo2recs
Memory for compression 0b, decompression 0b, cache 0b
Compressed 1 file, 1,811,671,405 => 3,275,785,748 bytes. Ratio 180.82%
Compression time: cpu 2.70 sec/real 145.77 sec = 2%. Speed 12.43 mB/s
All OK

Testing archive: Data.arc
Testing time: cpu 4.84 sec/real 58.16 sec = 8%. Speed 31.15 mB/s
All OK

Most games use kraken so there is no need to pair it up with Selkie/Mermaid, in any case the scanner should help you pick

Can you try that input on Windows 10 please, there is something I need to confirm

Ok i can't understand how f*** is possible but...

On my computer where i have Windows Server 2016:

using oo2reck+oo2recs on the file game0.arc from Just Cause 4:

846MB -> 910MB

using the same tools but on my old pc with windows 10, intel core 2 duo quad, 8GB ram, HDD:

846MB -> 2.33GB

:confused:

Middle-Earth Shadow of War
 

Patch_01.arch06 406 MB > oo2core_5 > 1.11 GB

Creating archive: Data.arc using oo2reck
Compressed 1 file, 426,161,268 => 1,199,330,430 bytes. Ratio 281.43%
Compression time: cpu 0.63 sec/real 126.85 sec = 0%. Speed 3.36 mB/s
All OK

Testing archive: Data.arc
Testing time: cpu 0.56 sec/real 67.65 sec = 1%. Speed 6.30 mB/s
All OK

It indeed is confusing but it seems to only work properly on Windows 10 :(

I tested also "oo2recm" now on this old pc and works perfectly, so you are correct, doesn't work at all on windows server 2016.

Only one oodle library near compressor at the same time, right?

Could someone send me o5 + o6 libraries please? Thanks!

Yes its better to only have the required oo2core dll near the compressor or else you may face crashes or incomplete precompression. I have attached a zip file containing oo2core4-7.

I don't no, but I'll tried to pack game0.arc 846 MB, like L0v3craft

using oo2reck+oo2recs on the file game0.arc from Just Cause 4:
========================================
FreeArc 0.67 (March 15 2014) creating archive: data.arc
Compressed 1 file, 887,443,456 => 955,165,272 bytes. Ratio 107.63%
Compression time: cpu 1.00 sec/real 41.61 sec = 2%. Speed 21.33 mB/s
All OK

It turns out only 846=>910 MB
My PC:
Core i7-4720HQ 2.6 GHz, 16 GB RAM, Windows 10. P.S. On Windows 7 same 846=>910 MB

I don't understand how L0v3craft did 2.33GB

Good job Zee, you should enter some options for the choice of CPU threads for different hardware systems.

Same thing here on windows server 2016. If i run the tools on my old pc with windows 10 looks like the tools from razor are able to detect many more streams in the files, that is strange. Ok maybe it is not a problem of O.S. version but it is dependant from another thing, i don't know.

testing on AC Odyssey forge files 43.3gb inflate to 78730mb > srep+lolz 20gb
seems to me others that used afr get better result.
i need test with afr to be sure...
cpu usage like zig zag it took around 43min on I7 8700K at 4.7gb...

Update available

Changes

- Fixed compatibility issues... hopefully, give big thanks to ShivShubh and FitGirl for helping and community for tests

also work on monster world hunter inflate from 19.9gb to 34.4gb but i got blue sreen in the middle of lolz :eek:
my ssd nvme evo die after 378tbw :mad:

That's why you should never make your repacks on SSDs.

what do you recommend then, i know ssd's are bit fast when we use em to compress files but same "Dead"!

Always use HDDs for that, they have no write limits. In example, 378 TB of writes I exceed in a couple of weeks. Imagine I'd done it on SSD. SSDs are for OS or for rare read files. Never use them for frequent writes.

so basically, SSD's don't support frequent writing of the data.....
now my question is what should be the recommended speed of an HDD....i mean they differ in range of speeds and quality....that matters a bit