CLS-DiskSpan (UltraARC replacement)

[Razor12911]

I'll check it out

[BLACKFIRE69]

Originally Posted by Cesar82

Yes, I know that, but it's still great.
NORMAL: 1130 Kb (1.10 MB) >> UPX = 288 Kb
if compile with DELPHI 2010 (but not work) = NORMAL: 339 Kb >> UPX 119 Kb.
I will try with other dephi XE versions later.
But it would be useful to know what would be the minimum version of DELPHI that this source works for.
Thank you!

Cesar82,

I was able to reduce the size to 249kb with (Delphi Rio + UPX)


,

[Cesar82]

@BLACKFIRE69
I got smaller sizes using DELPHI 10 Seattle.
From XE8 all functions/classes are coded without any changes.
Using DELPHI XE4 through XE7 it is only necessary to include a variable to make the script compatible.
I tested the version compiled with DELPHI Seattle and it is working fine.
Only the same errors happen as in the original lib that Razor12911 shared.
I did not test these older versions XE4 to XE7, because as the Lib compiled with DELPHI Seattle was better than with previous versions it does not make sense to test.


Adaptation of DiskSpan R3 to work with old delphi (Delphi XE4...XE7)

Changhed this:

begin
  FormatSettings := TFormatSettings.Invariant;

Changed To:

var
  FormatSettings: TFormatSettings;
begin
  GetLocaleFormatSettings(LOCALE_SYSTEM_DEFAULT, FormatSettings);

I think that this would not bring errors in the use of the library.
But as the size of the library compiled with DELPHI Seatle is smaller, it is more viable to use it.
This other executable compressor used from the image "uPACK" I found the compression rate very good and super fast, but in some libs it brings false positives.

[BLACKFIRE69]

Originally Posted by Cesar82

@BLACKFIRE69
I got smaller sizes using DELPHI 10 Seattle.
From XE8 all functions/classes are coded without any changes.

Adaptation of DiskSpan R3 to work with old delphi (Delphi XE4...XE7)

Changhed this:

begin
  FormatSettings := TFormatSettings.Invariant;

Changed To:

var
  FormatSettings: TFormatSettings;
begin
  GetLocaleFormatSettings(LOCALE_SYSTEM_DEFAULT, FormatSettings);

Cesar,

I compiled it in Delphi Seattle without changing and changing the code, but it wasn't working.

[Cesar82]

Originally Posted by BLACKFIRE69

Cesar,
I compiled it in Delphi Seattle without changing and changing the code, but it wasn't working.

Thanks for the info.
There's no logic (I don't understand why Seattle isn't even that old).
Here it was working when I tested it before (I only tested the extraction using CIU).
I didn’t even remember to try to pack it with him because the idea was to include it in CIU to perform the extraction.
I'm going to do some tests in a few days.

[Cesar82]

@Razor12911
If possible, look at the attached file.
The native methods of freearc "2", "3", "4", "5" do not work if you include cls diskspan R2/R3.
Or am I setting the command line wrong?
P.S. If I remove the diskspan from the command line it compresses normal.
P.S. The other native methods "0", "1", "9x -ld192m", "x -ld800m", "x -ld1600m" work normally.
Thank you!

[Razor12911]

Originally Posted by Cesar82

@Razor12911
If possible, look at the attached file.
The native methods of freearc "2", "3", "4", "5" do not work if you include cls diskspan R2/R3.
Or am I setting the command line wrong?
P.S. If I remove the diskspan from the command line it compresses normal.
P.S. The other native methods "0", "1", "9x -ld192m", "x -ld800m", "x -ld1600m" work normally.
Thank you!

have you tried to use other cls other than diskspan to see how it reacts?
1.PNG

2.PNG

3.PNG

the cls by itself...

4.PNG

perhaps this is why I have never mixed default freearc methods with additional methods set by user (-m1..9,x that is)

[Cesar82]

Originally Posted by Razor12911

have you tried to use other cls other than diskspan to see how it reacts?

Do not.
I just tested how I sent the file and realized that from method 2..5 it showed the same error as the third image.
I found it strange to support method 0 and 1 and the others display error.
See the result in the VIDEO of my test using only CLS-DiskSpan.dll + Arc Method

Do you mean that FreeArc does not support combining methods 2 - 5 with CLS?

I wanted to enable users to use these native methods when using diskspan.bat that I changed.
I came to think that it could be some config of the arc.ini that I used so I took Arc.ini and arc.groups directly from the freearc folder and even so when using cls-diskspan it showed errors and without it it does not display an error.
Thanks.

I ran some tests and found that:
If you open a compressed file with FreeArc and go to Info, only with the native method 2..5 it is possible to obtain the combination of the method referring to preset 2..5.


If you include these methods as presets of 2..5 in Arc.ini, the methods are now supported by Cls DiskSpan


Just adding the Arc.ini presets 2..5 makes it possible to support methods using cls-diskspan.

[Compression methods]

2=rep:1kb:256+exe+4x4:tor:1kb:h4kb
3=rep:1kb+exe+delta+4x4:lzma:32kb:fast:32:mc4
4=rep:1kb+exe+delta+4x4:lzma:32kb:normal:16:mc8
5=rep:1kb+exe+delta+4x4:lzma:32kb:normal:32:mc32

After opening the compressed file with FreeArc it is possible to verify that the method was used correctly.

[Razor12911]

I was going to say, that even using these

Arc.exe a -cfgArc.ini -ep1 -r -ed -s; -w.\TEMP -m2+rep -dp".\files" ".\Data2.bin.001"
Arc.exe a -cfgArc.ini -ep1 -r -ed -s; -w.\TEMP -m3+rep -dp".\files" ".\Data3.bin.001"
Arc.exe a -cfgArc.ini -ep1 -r -ed -s; -w.\TEMP -m4+rep -dp".\files" ".\Data4.bin.001"
Arc.exe a -cfgArc.ini -ep1 -r -ed -s; -w.\TEMP -m5+rep -dp".\files" ".\Data5.bin.001"

causes the same errors so it's not just cls issue.
I had to look at Freearc source and to give proper aliases for m2-5, you should look at this

5.PNG

-m2 is actually -mrep:96m:256:c256+exe+tor:6

use this to fill up the rest

[Cesar82]

@Razor12911
Thanks for the info, but I don't understand this FreeArc souce language (I was even more confused).
You could share the native presets (0..5 and "9x -ld192m", "x -ld800m", "x -ld1600m" (UltrArc GUI predifinitions)) so that I can include them in the [Compression Methods] section of Arc.ini.
I don't know if I can use memos numbers as a preset (Only with DiskSpan using 2..5 in the presets it worked here).

Currently I used the "Arc.ini" file for DiskSpan.bat "(UltraARC 2900 R8 + DiskSpan.bat)".
Apparently it's working well (I don't know if it's the right way)
In this way the user can use an option such as "3" or "FA-FAST" (optional) that will have the same result.

[Compression Methods]
FA-STORE=0
FA-MIN=1
FA-LOW=2
FA-FAST=3
FA-NORMAL=4
FA-HIGHT=5
FA-BEST=9x -ld192m
FA-MAX=x -ld800m
FA-ULTRA=x -ld1600m

2=rep:1kb:256+exe+4x4:tor:1kb:h4kb
3=rep:1kb+exe+delta+4x4:lzma:32kb:fast:32:mc4
4=rep:1kb+exe+delta+4x4:lzma:32kb:normal:16:mc8
5=rep:1kb+exe+delta+4x4:lzma:32kb:normal:32:mc32

[Razor12911]

Originally Posted by Cesar82

Thanks for the info, but I don't understand this FreeArc souce language (I was even more confused).
You could share the native presets (0..5 and "9x -ld192m", "x -ld800m", "x -ld1600m" (UltrArc GUI predifinitions)) so that I can include them in the

don't be lazy bro

5.PNG

# is a number, like -m2
if one uses 2, it then triggers the next part #rep+exe+#xb
then you check what 2rep+exe+2xb is
2rep is rep:96m:256:c256
2xb is tor:6
then you have rep:96m:256:c256+exe+tor:6

is the same when you use 5, it's not that difficult to follow.

when # is 5, we have 5rep+exe+5xb
5rep is rep:96m
#xb for 5 is delta + #binary

we check what 5binary is
5binary is lzma:96m:normal:32:mc32

then you combine everything
rep:96m+exe+delta+lzma:96m:normal:32:mc32

m9x follows this #x
where #x = #xb/#xt
9x= 9xb =delta + 9binary = delta+lzma:254m:max
#xt is a mask for text ($text)
9xt=dict:128m:75% + 9binary = dict:128m:75%+delta+lzma:254m:max

not sure about the 75% part of dict but that is what the method says

I also don't know this programming language either but I understood it just by looking at it, it's not that hard to parse the other methods.

[Cesar82]

Originally Posted by Razor12911

# is a number, like -m2
if one uses 2, it then triggers the next part #rep+exe+#xb
then you check what 2rep+exe+2xb is
2rep is rep:96m:256:c256
2xb is tor:6
then you have rep:96m:256:c256+exe+tor:6

I understood the line of reasoning.
I just don't understand why the final file has different compression following this line of reasoning.

Where can I get this freearc src?

[Razor12911]

Originally Posted by Cesar82

I understood the line of reasoning.
I just don't understand why the final file has different compression following this line of reasoning.

looks like I missed this part, you can just go ahead and write xtor:5 instead of tor:6
7.PNG
8.PNG

Where can I get this freearc src?

it's part of this project
https://krinkels.org/resources/fa_protect.5/

[BLACKFIRE69]

CLS-DiskSpan v2.0 - Overview & What's New

A reliability + performance overhaul of the FreeArc disk-spanning plugin.
This is a matched-pair release: archives written by v2.0 are unpacked by v2.0.
The on-disk volume format changed (clean break) - v2.0 will NOT read volumes
produced by the old version, and the old version will NOT read v2.0 volumes.

0. WHAT IT IS

v2.0 is a big reliability + speed overhaul of Razor12911's original plugin.

  Technical Specifications
    Built with ..... RAD Studio 13.1.
    Platform ....... Windows, 32-bit.
    Targets ........ FreeArc 0.67 (March 15 2014).
	
    Disc format .... self-describing v2 volumes: a "set GUID" shared by every
                     disc of one archive, plus a CRC32 on every disc.
					 
    I/O buffer ..... 1 MB by default; override with the CLS_BUFKB environment
                     variable (in KB, clamped 64 KB .. 64 MB).
					 
    Debug log ...... OFF by default; set environment variable CLS_DEBUG_LOG=1 
	                 to append diagnostics to cls-diskspan.log next to the DLL.

  Credits / Authors
    Original ....... Razor12911
    v2.0 overhaul .. BLACKFIRE69

  Distribution Files
    cls-diskspan.dll  REQUIRED. The plugin. Drop it next to Arc.exe.
    Arc.exe           REQUIRED. FreeArc's command-line driver (ships in bin\).
    oscdimg.exe       Optional. Only for "--makeiso"; must sit next to Arc.exe.
    arc.ini           Optional. FreeArc method config (example in bin\).
    arc.groups        Optional. File-grouping masks (example in bin\).

  Exported API Functions
    ClsInit        The host hands over the source path + a parent-window
                   handle (used by Inno Setup installers).
    ClsFunc1       (ClsSourcePath) the host sets/updates the folder to read the
                   sibling discs (2, 3, ...) from.
    ClsFunc2       (ClsDiskRequest) the host installs a callback diskspan can
                   call to ask the user to "insert the next disc".
    DiskSpanInfo   Extra (not called by FreeArc): returns the build +
                   format-version + copyright string so any tool (or the
                   "arc.exe --version" command) can identify the DLL.

1. WHAT'S NEW IN v2.0

- It just works for EVERY layout. 
   1. Solid (-s;) OR non-solid (-s-), 
   2. Big OR small solid blocks (-s<size>), 
   3. and file-mask / grouped packs (arc.groups) 
   
  all span and round-trip correctly. 
  The old "the archive must be fully solid" limitation -
  where anything else packed fine but silently produced broken discs - is GONE.

- Mixed group sizes are flagged. With file-mask groups, all groups share ONE
  volume set, so it can only use a single disc size - the first block's (and the
  first block is chosen by FreeArc, not diskspan). If another group asks for a
  different diskspan:first:other size, diskspan keeps the first and warns once
  (the set still packs and unpacks fine). Use the same size on every group.
  
    eg: -s; -m.../$grprpf=precomp+diskspan:10mb:10mb/$grpbin=lzma+diskspan:20mb:20mb

- Damage is caught. Every disc now carries a CRC32 checksum and shares a "set
  GUID". A truncated, byte-flipped, swapped, or foreign disc (one from a
  different archive) is detected and reported instead of silently producing a
  bad install.

- No more hangs or crashes. A missing disc during a silent/hidden run fails
  cleanly with a non-zero exit (it never freezes on a hidden dialog), 
   
   eg: Arc.exe x -o+ -dpOutput\ disk\game-1.bin < nul
   ("< nul" forces the non-interactive path)
  
  and no Delphi error can crash back into FreeArc - failures map to 
  proper CLS error codes.

- Safe inside installers. The --sort/--makeiso/--version helpers only run under
  FreeArc's Arc.exe; embedded in an Inno Setup installer (unarc) the DLL
  never touches the host process, so it can't accidentally close the installer.

- You see the REAL reason a span failed. On a bad decode FreeArc otherwise only
  prints its own misleading line ("output block size in (de)compressMem is not
  enough ..."). diskspan now prints the true cause first, on its own line.
  - e.g. "diskspan: missing volume game-3.bin" or "diskspan: CRC mismatch ...".

- An archive kept in a subfolder extracts correctly. The sibling discs are
  found next to the main archive, not in the current directory - so
  "arc t out\game-1.bin" works (the real conversion flow packs into a subfolder).

- The disc-tools are reliable.
    --sort     lays the discs out into burnable DISC_1, DISC_2, ... folders
               (with autorun.inf + setup files) and now exits cleanly.
    --makeiso  turns those folders into .ISO images and no longer freezes; a
               folder is deleted ONLY after its ISO is safely written, so a
               missing/failing oscdimg.exe can never destroy your discs.

- Flexible disc naming, auto-detected from the name you choose - no new option:
    game-1.bin    ->  game-2.bin,  game-3.bin      (classic digit step)
    data.bin.001  ->  data.bin.002, data.bin.003   (numbered extension)
    setup-1a.bin  ->  setup-1b.bin ... 1z, 1aa     (letters per disc, number
                       fixed - great for game collections)

					   
  Just put a number in the name (game-1.bin, not game.bin). If you forget, it
  still works, but the extra discs get plain names (game_002.bin, ...) and you
  get a short reminder at pack time.

- Locate-a-missing-disc. In an interactive run (an installer window OR a plain
  console "arc x"), a Browse dialog lets the user point to a disc that is not
  where it was expected, then the extract continues. Silent/hidden/redirected
  runs show nothing and just fail cleanly - they are never blocked.

- It's faster. The per-disc checksum was rewritten (slicing-by-8) and I/O is
  done in 1 MB chunks (tunable, CLS_BUFKB), cutting the spanning overhead
  from roughly +96% over a raw store to about +20%. At real multi-GB disc sizes
  the compressor dominates anyway, so spanning is effectively free.

- Version probe. "arc.exe --version" prints the build, disc-format version and
  copyright; the DiskSpanInfo() export returns the same string to tools.

2. ORIGINAL vs MODIFIED (v1 -> v2.0)

╭───────────────────────────┬───────────────────────────┬──────────────────────────────╮
│ Area                      │ Original (v1)             │ Modified (v2.0)              │
├───────────────────────────┼───────────────────────────┼──────────────────────────────┤
│ > Solid archives          │ OK                        │ OK                           │
│ > Non-solid (-s-)         │ SILENT CORRUPTION         │ OK                           │
│ > Small blocks (-s<size>) │ SILENT CORRUPTION         │ OK                           │
│ > File masks / arc.groups │ unsupported / corruption  │ OK                           │
│ > Mixed group sizes       │ SILENT CORRUPTION         │ OK (first size wins + warns) │
│ > Archive in a subfolder  │ extract could fail        │ OK                           │
│ > Per-volume integrity    │ none                      │ CRC32 (header + payload)     │
│ > Wrong/foreign volume    │ used blindly              │ rejected (set GUID)          │
│ > Missing volume (hidden) │ could hang on a dialog    │ clean nonzero exit           │
│ > Missing volume (UI)     │ host prompt only          │ browse (UI or console)       │
│ > Failure reason shown    │ FreeArc generic only      │ + diskspan's real reason     │
│ > Disc-1 size budget      │ per solid block           │ shared across all blocks     │
│ > Error across DLL bounds │ could crash FreeArc       │ mapped to CLS error codes    │
│ > Volume naming           │ digit substitution only   │ Type 1 / 2 / 3 (auto)        │
│ > Non-numbered name       │ silent _NNN fallback      │ warns at pack time           │
│ > --sort exit code        │ wrong ("unknown command") │ clean exit 0                 │
│ > --makeiso               │ hung after the first ISO  │ builds all, safe delete      │
│ > CRC speed               │ n/a (no CRC)              │ slicing-by-8 (~20% overhead) │
│ > Copy buffer             │ 64 KB                     │ 1 MB (tunable, CLS_BUFKB)    │
│ > Version/copyright probe │ none                      │ --version + DiskSpanInfo()   │
│ > Build toolchain         │ older Delphi              │ RAD Studio 13.1, Win32       │
│ > Compiler warnings       │ several                   │ zero (clean build)           │
╰───────────────────────────┴───────────────────────────┴──────────────────────────────╯

  Important: v1 and v2.0 use DIFFERENT on-disk disc formats. Always pack AND
  unpack with the SAME version - v2.0 cannot read old discs, and vice-versa.

3. QUICK START EXAMPLES

Run these from the folder that holds Arc.exe + cls-diskspan.dll. "-w" sets a
temp folder; "diskspan:<first>:<other>" is first-disc size : every other disc's
size. Put a NUMBER in the archive name (game-1.bin).

  * Show the build + copyright:
    > Arc.exe --version

  * Pack + split, one solid block, no compression (fast smoke test):
    > Arc.exe a -ep1 -r -ed -s; -w.\temp -m0+diskspan:2mb:2mb out\game-1.bin "_data\*"

  * Pack + split with REAL compression (put diskspan LAST in the chain):
    > Arc.exe a -ep1 -r -ed -s; -w.\temp -msrep+lzma+diskspan:4467mb:4474mb out\game-1.bin "MyFiles\*"

  * Pack + split, NON-solid (-s-, every file its own block - the old silent-
    corruption case, now correct):
    > Arc.exe a -ep1 -r -ed -s- -w.\temp -m0+diskspan:2mb:2mb out\game-1.bin "_data\*"

  * Pack + split, SMALL solid blocks (-s<size>, e.g. 8 MB blocks):
    > Arc.exe a -ep1 -r -ed -s8m -w.\temp -m0+diskspan:2mb:2mb out\game-1.bin "_data\*"

  * Pack + split, file-MASK groups (arc.groups defines $grpbin etc.; end each
    group's method with diskspan so every group spans):
    > Arc.exe a -ep1 -r -ed -s; -w.\temp ^
      -mtor+diskspan:2mb:2mb/$grpbin=lzma+diskspan:2mb:2mb out\game-1.bin "_data\*"

  * Test the disc set (reads every disc + checks each disc's CRC):
    > Arc.exe t out\game-1.bin

  * Extract:
    > Arc.exe x -dpunpacked\ out\game-1.bin

  * Lay the discs out into burnable folders, then build one ISO per folder:
    > Arc.exe --sort    OutFolder DISC_ setup.exe setup.ico out\game-1.bin
    > Arc.exe --makeiso OutFolder\DISC_

  * Pick a naming style just by how you name the first disc:
    > Arc.exe a ... +diskspan:300mb:400mb data.bin.001 "MyFiles\*"   (-> .002, .003)
    > Arc.exe a ... +diskspan:300mb:400mb setup-1a.bin "MyFiles\*"   (-> 1b, 1c, ...)

  * Tuning / diagnostics (environment variables, optional):
    > setlocal & set CLS_BUFKB=4096   & Arc.exe ... <- (use a 4 MB I/O buffer instead of 1 MB)
    > setlocal & set CLS_DEBUG_LOG=1  & Arc.exe ... <- (write cls-diskspan.log next to the DLL)

 Ready-to-run versions of all of the above (double-clickable) are in
  bin\_examples\_demo\  - start with 00_version.bat.

.