Including and excluding files from data synchronization and online backup

Introduction

 You need to transfer files from one or more systems to a remote location for online backup, for sharing or for automating processes. This post explains how the includes and excludes work with EDpCloud file replication and online backup software. EDpCloud uses regular expressions to exclude (and exclude) file name and directory name patterns from being replicated an from being sent to the remote online backup or data transfer location.

Rules of including and excluding files and folders from synchronization

In order for a file or directory to be replicated to a remote or local site, the file name must match the includes pattern and must not match the excludes pattern.

Excluding files from data synchronization

When first installed, EDpCloud creates an excludes file named $ED_BASE_DIR/edpcloud/etc/excludes.
Users can tailor the excludes file to suite their need. The excludes file is the default and applies to all replication sets(links), to all senders and to all receivers in the configuration. A specific file for the replication set, for the synchronization file sender and for the data synchronization receiver can be specified using the keyword exfile=”filename_with_regex“;

Including files in the data synchronization

When EDpCloud is installed, there is no includes file in the system, hence the include pattern is the wild card (.*), which means match and include everything . Hence all files will be included if they do not match a pattern in the excludes files. Include patterns can be specified using the global file $ED_BASE_DIR/edpcloud/etc/includes, which applies to all file sync links, to all senders and to all receivers. An alternative is to use the keyword incfile=”fullpath_to_a_specific_includes_file“, which applies only to the replication set where, to the sender or to the receiver where it was specified.

Example: Using incfile and excfile to control what files are sychronized

Example of eddist.cfg

Use in the link/replication set section

…
link name=”l1″ incfile=”/usr/local/enduradata/edpcloud/inc1″ exfile=”/usr/local/enduradata/edpcloud/exc1″
…

Use in the file sender or receiver section

…
hname=”localhost” incfile=”/usr/local/enduradata/edpcloud/inc1″ exfile=”/usr/local/enduradata/edpcloud/exc1″
…

Example of regular expressions to include and exclude files from synchronization and online backup.

/usr/local/enduradata/edpcloud/inc1 content:

^/home/.*$
^/data/.*$

/usr/local/enduradata/edpcloud/exc1 content:

^/etc/.*
.*/core$
^/usr/.*
….

It is critical to know how to use the includes and excludes, especially if you decide to replicate data
to “/” under various *UNX* flavors.

More information about regular expressions can be found in the following links:

https://www.gnu.org/software/findutils/manual/html_node/find_html/posix_002dextended-regular-expression-syntax.html

https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap09.html

Additional EDpCloud documentation

See eddist.cfg man page or html document under $ED_BASE_DIR/edpcloud/doc for additional information.

Download EDpCloud and Start Synchronizing Between Different Systems Now.

https://www.enduradata.com/data-protection-mac-windows-linux-solaris-aix/

https://www.enduradata.com/bidirection-file-replication-configuration-linux/

Call 952-746-4160 for more information.