FileForums - Best Compression Methods for 'Specific' Games. Q&A

I'm very new to lzo1x pre-compression, could anyone explain why XTool in this example is able to detect some streams but not able to process any? I've tried levels 1 - 10 and the same amount of streams are detected but none are able to be processed

Code:

Creating Archive Please Wait...



FreeArc 0.67 (March 15 2014) updating archive: Data.arc

Compressing 1 file, 38,255,712 bytes. Processed   0%

Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 0/2854

Time: 00:00:01 (00:00:03)

Memory: 128 MB (128 MB)



100%

Errorlevel=0

Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%

Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s

All OK



All done.

The algorithm used is definitely lzo1x, and creating a database with bms2xtl detects an additional ~5000 streams but alas none are detected. I've also not been able to find many other lzo libraries to test with so if its a library issue, if anyone could point me in the direction to look for others it'd be greatly appreciated.

Lord.Freddy

19-09-2022 23:29

Quote:

Originally Posted by L33THAK0R (Post 498393)

Code:

Creating Archive Please Wait...



FreeArc 0.67 (March 15 2014) updating archive: Data.arc

Compressing 1 file, 38,255,712 bytes. Processed   0%

Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 0/2854

Time: 00:00:01 (00:00:03)

Memory: 128 MB (128 MB)



100%

Errorlevel=0

Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%

Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s

All OK



All done.

maybe you need to remove the level option, sometimes it works.

L33THAK0R

20-09-2022 00:10

Quote:

Originally Posted by Lord.Freddy (Post 498395)

maybe you need to remove the level option, sometimes it works.

Unfortunately seems like some data actually got stripped away doing that, resulted in a 99.51% ratio :(

Edison007

20-09-2022 08:01

Quote:

Originally Posted by L33THAK0R (Post 498393)

Code:

Creating Archive Please Wait...



FreeArc 0.67 (March 15 2014) updating archive: Data.arc

Compressing 1 file, 38,255,712 bytes. Processed   0%

Compressing 38,255,712 bytes with "C:\Compressor_Resources\XTool_2020\xtool_0.6.5\XTool.exe" precomp -mlzo1x:l6 -c512mb -t95p $$arcdatafile$$.tmp $$arcpackedfile$$.tmp

XTool is created by Razor12911



Streams: 0/2854

Time: 00:00:01 (00:00:03)

Memory: 128 MB (128 MB)



100%

Errorlevel=0

Compressed 1 file, 38,255,712 => 38,255,982 bytes. Ratio 100.00%

Compression time: cpu 0.05 sec/real 1.63 sec = 3%. Speed 23.44 mB/s

All OK



All done.

Maybe used pro-functions. Upload example file for testing

L33THAK0R

21-09-2022 03:35

2 Attachment(s)

Quote:

Originally Posted by Edison007 (Post 498398)

Maybe used pro-functions. Upload example file for testing

Thanks for offering to have a looksie, really appreciate any insight you can offer, also in case its of any interest I've also got an LZ4 sample I've been having similar issues with, tried about a dozen libraries but no dice so far, xtool (0.6.5) is also detecting streams but unable to pick any up unfortunately.

KaktoR

21-09-2022 04:09

For your lz4 sample you have to use lz4f with level 2 and pick the right lz4 library

Code:

Compressed 1 file, 32,882,867 => 158,560,939 bytes. Ratio 482.20%

Compression time: cpu 0.08 sec/real 1.18 sec = 7%. Speed 27.85 mB/s

All OK

You have to look which level produces the closest recompressed size to the original size

https://i.imgur.com/xXoiZtm.png

However in a quick test there are still some streams left behind.

L33THAK0R

21-09-2022 07:38

Quote:

Originally Posted by KaktoR (Post 498403)

For your lz4 sample you have to use lz4f with level 2 and pick the right lz4 library

Code:

Compressed 1 file, 32,882,867 => 158,560,939 bytes. Ratio 482.20%

Compression time: cpu 0.08 sec/real 1.18 sec = 7%. Speed 27.85 mB/s

All OK

You have to look which level produces the closest recompressed size to the original size

https://i.imgur.com/xXoiZtm.png

However in a quick test there are still some streams left behind.

Ah didnt realise I had to specify lz4f, many thanks, also what tool are you using to get that console output if you dont mind me asking?

KaktoR

21-09-2022 08:04

--verbose mode from xtool

Edison007

21-09-2022 08:38

1 Attachment(s)

Quote:

lzo_sample.rar

The old function LZO1X-1 was used. You can use these libraries. But not all streams are successfully recompressed, because lzo-detect in xtool is't ideal.

L33THAK0R

21-09-2022 10:48

Quote:

Originally Posted by Edison007 (Post 498410)

The old function LZO1X-1 was used. You can use these libraries. But not all streams are successfully recompressed, because lzo-detect in xtool is't ideal.

Just ran a quick test, this works wonderfully, thanks so much! I've had this sitting on the back-burner for months, it's very satisfying to have it finally sorted!

L33THAK0R

23-09-2022 01:34

Does anyone know if theres a way to bulk apply/insert a hexadecimal pattern into a given chunk of data if given a start and end offset? I've got some projects I'm working on that require a few hundred blanking/0-filling operations and its awfully slow to do so with HxD's GUI. I've gone through the SFK documentation a half dozen times and alas to no avail. My current solution is just to have a massive blank file and drawn on the given offsets from it but this is obviously not very practical and is a pain having to increase its size should I be attempting to patch out a chunk from a larger archive in the future.

KaktoR

23-09-2022 01:43

Try with xtool's erase/replace command

dixen

23-09-2022 04:34

Pathfinder: Kingmaker (GOG Version) [New repack]

Bundle Folder (excluded manifest files)
XTool v0.65 (unity+lz4 plugin v1.75)+srep+lolz
21 gb > 50 gb > 25 gb > 7.2 gb

Sound files (*.wem & *.bnk)
srep+lzma2
3.2 gb > 2.6 gb

*.assets *.resS & extensionless files
srep+lolz
13.4 gb > 2.8 gb

Rest files
400 mb > 65 mb

Repack's size - 12.7 gb
Install time - 29 min. (R5 3600)

shazzla

24-09-2022 09:44

Quote:

Originally Posted by L33THAK0R (Post 498404)

Ah didnt realise I had to specify lz4f, many thanks, also what tool are you using to get that console output if you dont mind me asking?

Hi Kaktor and everybody !

What is this tool in the screenshot ? And what parameter was used?
Im out nearly 2 years from repacking!
Btw anyone can share me an lz4 library collection for xtool?
Many thanks! :)

KaktoR

24-09-2022 09:51

1 Attachment(s)

Quote:

Originally Posted by shazzla (Post 498466)

xtool --verbose mode

Attached file contains some lz4 libraries

All times are GMT -7. The time now is 23:50.

Page 211 of 312