Go Back   FileForums > Game Backup > PC Games > PC Games - CD/DVD Conversions > Conversion Tutorials

Reply
 
Thread Tools Display Modes
  #1  
Old 26-03-2018, 08:07
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 264
Thanks: 190
Thanked 323 Times in 118 Posts
elit is on a distinguished road
xnLZ - parallel lzma with entropy scan intended as 64bit 4x4:lzma replacement

xnLZ is a multi-threaded lzma (de)compressor heavily based on plzip, of which it is fork of and NOT compatible with anymore.
Its main purpose is to offer alternative to existing 4x4:lzma chain(also known as xlzma) in FreeArc(or other archivers that
can utilize external compressors), but with 64bit support and same cool features that made 4x4 so great. Although xnLZ was
intended to be used for <stdio> mode with FreeArc, it can also work on its own with files, or can be chained with other tools
that support external compressors.

Here are the main features:

- xnLZ utilize entropy check that scan incoming data blocks during compression and can copy them directly if they are not
compresseable enough. All this can be defined by user. This means some data that do not benefit from typical lzma compression
can bypass compress routine completely, speeding whole process. On combined data sets this could usually be anywhere
between 50-300% speedup and decompression speed will benefit as well.

- Data blocks that are copied directly still have their CRC attached and are verified during test or decompression.

- xnLZ was designed to utilize all CPU cores during decompression through <stdio> for maximum speed. This feature actually
existed(as of time of fork at least) in original plzip in complete infrastructure, all that was needed was to increase
buffer/packet limit, which was artificially hardcoded to low value. Now user can directly define how much buffer to allocate,
but it comes at the cost of utilizing more memory during decompression. As example, to fully utilize 4 core CPU with data
blocks of 256MB, ~1300MB-2GB may be needed during decompression, depending on settings. In my tests I was able to load my
CPU fully with slots value 128 on data blocks of 256MB size, which used only around ~1GB RAM.

- xnLZ use hardcoded lzma:lc(literal context) parameter set to 8 instead of default 3 for maximum compression.
- xnLZ expose additional lzma:mc(matchfinder cycles count) option for the user to set, which tend to have big impact on compression speed
- Not dependent on Cygwin libraries anymore

More info here:
https://encode.ru/threads/2932-xnLZ-...6404#post56404

Quick random benchmark on combined data folder with lot of zip files, few big exes and dlls:
Code:
xlzma(4x4:lzma) & xnlz = d64m, b64m, fb32, mc32, E99% <- both use lzma1:lc8
7z 16.04     = 0=lzma2:d=64m:fb=32:lc=4:mc=32         <- use lzma2

orig data: 3.24g


xlzma(4x4)- c 1:15m, d 5s,  2.76g
xnlz      - c 1:40m, d 13s, 2.74g
xnlz:E98  - c 1:23m, d 11s, 2.76g
7z        - c 8:12m, d 26s, 2.70g
xnlz:E100 - c 6:15m, d 49s, 2.71g
EDIT: here is the result on uncompressed data:
Code:
orig file: 660.3mb
7z: c 1:15m, d 12s, 270.19mb
xnlz: c 1:17m, d 7s, 270.85mb
The reason why xnlz was slower during decompression in first test when entropy skipping was disabled and despite using all cores was because archive contained compressed data(zip files) and entropy skipping was disabled. Lzma2 was quicker because it does better job handling compressed data than lzma1. However, first it is no match on whole block entropy skipping in terms of speed when compared to xlzma or xnlz, and second here it was finally slower with inflated data decompression.
Attached Files
File Type: 7z xnLZ-1.0.7z (424.2 KB, 266 views)

Last edited by elit; 26-03-2018 at 08:24.
Reply With Quote
The Following 7 Users Say Thank You to elit For This Useful Post:
78372 (26-03-2018), Andu21 (26-03-2018), artag (28-03-2018), EzzEldin16 (26-03-2018), KaktoR (26-03-2018), mubbii (17-11-2018), Simorq (26-03-2018)
Sponsored Links
  #2  
Old 02-04-2018, 11:57
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 264
Thanks: 190
Thanked 323 Times in 118 Posts
elit is on a distinguished road
This is xnlz with lzma:pb4 (and lzma:lc8 like previous release). This tend to give slightly better compression in games(0.5-2%) from what I have seen. However, this and last release are not compatible, keep that in mind. I plan to stick with this setup, therefore its better to release now than later when people adopt it for too long.
Sources are same only change was "pos_state_bits" param located in LZ dir in xnlz.h file. You can also test it and let me know if it improved ratio for you.

xnlz-pb4-fix.zip

Last edited by elit; 02-04-2018 at 18:42.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Simorq (02-04-2018)
  #3  
Old 02-04-2018, 12:55
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,304 Times in 464 Posts
Simorq is on a distinguished road
xnlz.exe has stopped working
Reply With Quote
The Following User Says Thank You to Simorq For This Useful Post:
elit (02-04-2018)
  #4  
Old 02-04-2018, 18:43
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 264
Thanks: 190
Thanked 323 Times in 118 Posts
elit is on a distinguished road
Quote:
Originally Posted by Simorq View Post
xnlz.exe has stopped working
Oh, I forgot to change gcc flags, I compiled it with -march=native which is for Haswell. I re-uploaded more general one(-mtune=native -msse2) hopefully it work now, my apology.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Simorq (02-04-2018)
  #5  
Old 03-04-2018, 06:59
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 264
Thanks: 190
Thanked 323 Times in 118 Posts
elit is on a distinguished road
Pls let me know how it fare against original one especially in games, compress few and post, if it shows continuous gain and/or only minimal loss occasionally this will be permanent standard setting for future.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Simorq (15-04-2018)
  #6  
Old 06-04-2018, 09:37
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 264
Thanks: 190
Thanked 323 Times in 118 Posts
elit is on a distinguished road
So I made more tests on multiple data. My conclusion is that its better to stick to original one, it is very good already and lzma:pb4 did hurt a bit too often, even if only negligible. Difference was tiny anyway.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Simorq (15-04-2018)
  #7  
Old 15-04-2018, 13:18
Simorq's Avatar
Simorq Simorq is offline
Registered User
 
Join Date: Mar 2014
Location: Iran
Posts: 642
Thanks: 3,602
Thanked 1,304 Times in 464 Posts
Simorq is on a distinguished road
x86 compiled?
Reply With Quote
  #8  
Old 16-04-2018, 16:11
elit elit is offline
Registered User
 
Join Date: Jun 2017
Location: sun
Posts: 264
Thanks: 190
Thanked 323 Times in 118 Posts
elit is on a distinguished road
Quote:
Originally Posted by Simorq View Post
x86 compiled?
Simorq you mean 32bit? You can compile under msys2 if you need it, use -m32 in makefile instead -m64. But what would be the point? Its IMO better to use internal FA's lzma them.
Reply With Quote
The Following User Says Thank You to elit For This Useful Post:
Simorq (17-04-2018)
  #9  
Old 03-01-2025, 07:15
kj911 kj911 is offline
Registered User
 
Join Date: Apr 2010
Location: world
Posts: 186
Thanks: 136
Thanked 64 Times in 45 Posts
kj911 is on a distinguished road
Bumping the old thread....

Is the lp parameter fixed to lp0?? Or can it be parameterized? In many cases, the lp0:pb0/pb2 pair is the best, including lc8, and values ​​between pb4 and lp1-4 are rarely seen in compressed archives. With DLZ, the lp/pb switches are chained together, where lp0 the best works. Does the mc switch support max. 1023/1024 or more? Just like, the maximum value of fb is only 273? It's strange, it's not often that you see a method like xnLZ used among the finished conversions, instead of the classic 4x4:lzma.

The x86 edition would make sense so that when decompressing we are not limited to using Vista/Win7 x64 or newer OS, it could also be installed on 32bit XP, although there is also the ARC built-in xlzma method or DLZ.

Last edited by Joe Forster/STA; 03-01-2025 at 07:42. Reason: parameter was converted into smiley
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
plzip_modded - parallel 64bit lzma (de)compressor and a direct 4x4:lzma replacement elit Conversion Tutorials 17 26-03-2018 08:08
Company of Heroes sylar91 PC Games - Protection List 2 01-07-2011 05:31



All times are GMT -7. The time now is 01:26.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2025, vBulletin Solutions Inc.
Copyright 2000-2020, FileForums @ https://fileforums.com