Exdupe with stdout ?!

[shazzla]

Hi guys !

Few days ago i tried to use Exdupe with stdout in arc.ini ,but i failed.
Anyone succeeded with this ?

Yes ,i know,there is SREP. Its okay,but in some case,it eats insanely hughe amount of RAM,even with -dngb parameter.

Any help appreciated !
Thanks.
(Atm away from PC,im on tablet)

[Bulat]

increase -l parameter in srep, it should be always better than exdupe

[shazzla]

Thank Bulat!

I know about the - l - parameter ,but it doesnt decrease the decompression memory req. Only a little bit!
Real example :
Approx 40 Gb dataset with L512 ,decomp memory 4.8 Gb
l1024-l4096 ,decomp memory around 3.9 gb.
-dngb n=1-4 ,M3F-M5F used in both case !
Those decomp.mem.reqs are too much for me.
Thats why i want to use exdupe without tempfile.
Btw i love SREP...

Any idea to decrease decomp mem reqs?

[Bulat]

-m0 -d1g will definitely use less than 1 GB

[78372]

:confused: m0 is the rep method, so it's memory usage is obviously low.


BTW, in case of x86 srep, how can we prevent srep from creating virtual temp file(I have only 4GB RAM)?

[FD_RLT]

:confused: m0 is the rep method it's memory usage is obviously low.


BTW, in case of x86 srep, how can we prevent srep from creating virtual temp file(I have only 4GB RAM)?

Why you don't want it to create virtual temp file if you only have 4 GB RAM ?

[78372]

Because of the massive read and writes in hdd

[shazzla]

uses razor's CLS-SREP.DLL.
it uses memory for TMP ,but u need to keep decomp memory req within considerable values.

[78372]

That's the problem. I know razor's cls is helpful, but hey it's only useful if you use 64bit srep because it allows you to use that. But the problem is, I use x86 editions and when you use that, you need the virtual memory.

Srep is a big problem for me when repacking games bigger as GTA V, because I use solid compression which requires a lot of ram to compress and decompress

[ChronoCross]

[External compressor:exdupe]
packcmd = exdupe.exe -x0 -m512 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = exdupe.exe -R $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
datafile = $$arcdatafile$$.tmp
packedfile = $$arcpackedfile$$.tmp

[shazzla]

thanks ,its okay ,but how can i use STDOUT ?

[ChronoCross]

thanks ,its okay ,but how can i use STDOUT ?
why you make so complicated?, go for the most simple way(freearc ->arc.ini)

7z.exe -an -ttar a -bd "D:\TEST\*" -so | exdupe.exe -x0 -m512 -stdin - -stdout | zstd.exe -f -19 -v - -o test.zst

[shazzla]

Sorry for the noob question,but how can i use this in arc.ini?
I mean pack and unpack command..

i tried several combination to pack/unpack. sadly without success. :(

Arc.ini PACK command :

packcmd = exdupe.exe -x1 -m512 -stdin -p -stdout <stdin> <stdout>

working on UNPACK......

[shazzla]

2 years passed and no success. Anyone ? :/

[shazzla]

hmmm..
tried many versions,but what is the unpack command for this ?

7z.exe -an -ttar a -bd "D:\TEST\*" -so | exdupe.exe -x0 -m512 -stdin - -stdout | zstd.exe -f -19 -v - -o test.zst

As the topic describes ,the main goal is use exdupe with stdio with freearc.
It would be nice.
Please ,anybody ?

[Razor12911]

I'll try to help but no promises okay? :)

[shazzla]

WoW !
Thank you very much !

[78372]

Does this support stdin/stdout at default or we need to make a patcher?

[shazzla]

In theory ,yes. It supports :

eXdupe 0.5.0 BETA. Copyright 2010 - 2013 eXdupe.com. UNSTABLE UNSTABLE

Full backup:
[-vxarctpmgl] [-f] [-s] <sources> <destination>
[-vxarctpmgl] <-stdin> <filename to assign> <destination>

<destination> can either be -stdout or a file. Use .full as file extension

Differential backup:
-D[vxarctpl] [-f] [-s] <sources> <.full file> <destination>
-D[vxarctpl] <-stdin> <filename to assign> <.full file> <destination>

<destination> can either be -stdout or a file. Use .diff as file extension

Restore full backup:
-R[vo] <.full file> <destination directory | -stdout> [files]
-R[vo] <-stdin> <destination directory>

[files] is one or more files, drives or directories to restore, typed as
provided by -L flag

Restore differential backup:
-RD[vo] <.full> <.diff file> <destination directory | -stdout> [files]
-RD[vo] <-stdin> <.diff file> <destination directory>

[files] is one or more files, drives or directories to restore, typed as
provided by -L flag

List contents:
-L <.full file | .diff file>

Flags:
-r Do not include sub directories
-o Overwrite existing files (default is to abort)
-c Continue if a source file cannot be read or does not exist (default is
to abort)
-xn Use compression level n for traditional data compression applied after
deduplication. Set n = 0 for none (lets you apply your own), x = 1 for
QuickLZ, x = 2 for zlib or x = 3 for bzip2 (default = 1)
-p Include named pipes
-l On *nix: Follow symlinks (default is to store link only). On Windows:
symlinks are always followed - storing link only is not supported
-a Store absolute and complete paths (default is to remove the common
parent path of items passed on the command line)
-s"d:" Use Volume Shadow Copy Service for local drive d: (Windows only)
-f"s" Filter files using a script, s, written in the Lua language
-gn Use n GB memory for a hash table (default = 1). Use -mn to specify
number of MB instead. Use 1 GB per TB data for best compression ratio
Differential backups will use the same memory as the full backup
-tn Use n threads (default = 8)
-vn Use verbose level n for n = 0...9 (default = 8)
-- Prefix items in the <sources> list with "--" to exclude them.

Quick example of backup, differential backups and a restore:
exdupe z:\database\ database.full
exdupe -D z:\database\ database.full database.diff1
exdupe -D z:\database\ database.full database.diff2
exdupe -RD database.full database.diff2 z:\database\restored\

More examples:
exdupe -stdin database.mdf -stdout < database.mdf > database.full
exdupe -m256t2 z:\vmdk\win\ z:\vmdk\mac\ z:\vmdk\hpux\ servers.full
exdupe -RD servers.full servers.diff z:\restored mac win
exdupe -c -f"return(dir or size < 1000000)" z:\stuff stuff.full
exdupe -s"c:" c:\ --c:\pagefile.sys system.full
exdupe -R data.full -stdout > data.xml

[felice2011]

The version recommended to be used is "eXdupe 0.3.5" configured in this way:

[External compressor:eXdupe]
packcmd = "EXDUPE\eXdupe.exe" -x3 -g1 -t2 $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = "EXDUPE\eXdupe.exe" -R $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
datafile = $$arcdatafile$$.tmp
packedfile = $$arcpackedfile$$.tmp

All other next versions create problems with FA;)

[Razor12911]

I tried to make a patch for this thing but it is half done. The main issue is that exdupe seeks while in stdio, kinda like razor compressor, it is a bit harder to patch something like this plus we have no idea how much it needs to reread from input.

If you test this via Freearc, it will fail but via Cmd it doesn't because Cmd allows seeking while FA doesn't , so if I have free time I'll probably make a full patch for it, a bit busy lately :(

[kj911]

eXdupe v1.0.0 Final released!

Joe: I know about it! Version 1.10beta1 and beta2 are already out. (BETA1 its bigger EXE than BETA2 and 1.0.0) Separately attached BETA1 version. Beta2 not tested in last time. Mini test results included compared to srep and srep+exdupe mixed precompression before via LOLZ.

https://github.com/rrrlasse/exdupe/raw/random/1.1.0.beta2/exdupe-beta.exe

UPD: 2023-11-14 - Results updated!

[Joe Forster/STA]

GitHub: https://github.com/rrrlasse/exdupe.

[Carldric Clement]

I did some full research on the eXdupe pre-process file. It looks like that was difficult to make it both stdin and stdout. So I did this to make it work by using the latest version of eXdupe like this:

[External compressor:exdupe]
header = 0
packcmd = exdupe -x0 -g1 -t2 $$arcdatafile$$.tmp -stdout <stdout>
unpackcmd = exdupe -R $$arcpackedfile$$.tmp -stdout <stdout>

Hope you guys can try if it works or not. :)