This article is about configuring Linux real-time bidirectional data replication and file synchronization (file sync) in EDpCloud software.
Introduction to Linux bidirectional replication for backup and file mirroring
Linux Bidirectional Replication Server
Our objective here is to back up and sync one Linux server to another and vice-versa.
Download Linux Bidirectional Replication Software
For example, if you modify file “foo” on a server called “svr1”, file foo will also be modified on server “svr2” and vice-versa(by sending only the deltas or the part of the file that changed).
You can also do the same thing I describe hereunder windows, but I will specifically discuss the Linux Bidirectional Replication here.
Without delay, let’s jump in and see how we can configure Linux real time bidirectional replication and filesyc to mirror file systems on servers or virtual machines.
Contact An Engineer Now
Please read this article first:
Requirements for Linux Bidirectional Replication and remote file mirroring
1. Download and install EDpCloud by visiting https://www.enduradata.com/downloads. If you have 64 bit Linux, download the 64-bit package; otherwise, download the x86 package (32 bit).
2. Install EDpCloud replication software
Details about the installation can be found in the articles cited above.
3. Have your list of directories or files you want to replicate handy: This is the list of directories you want to monitor in real time for backup and sync.
Configure Linux bidirectional data replication and file synchronization
1. cd $ED_BASE_DIR/etc
2. vi includes and add all the directories you want to include in the replication
The includes and excludes files use regular expressions. For more information, read the post on includes and excludes:
Excluding and including files from data synchronization.
Edit data replication includes file (You will find it in etc directory)
Your includes much have only the directories you want to be included. Everything is included by default if you don’t have an include file.
Add the following two lines using a text editor (or the browser-based GUI). This will allow the servers only to include the files that match the patterns below in the bidirectional data replication. Make sure you do the same on both servers.
^/data/
^/data/.*
This will only replicate and mirror files that are under /data and the directory /data itself.
Edit Linux replication excludes
Again see the post about excluding files from replication. You must exclude all directories such as etc, var, etc. The default excludes such a list. I cannot stress enough how important it is to use includes and excludes to avoid mirroring files in /etc, such as the passwords, hostnames, and IP addresses. Make sure you use the includes to specify what to replicate.
Install FUSE if your system does not have it already
EDpCloud requires FUSE. You can find the sources under $ED_BASE_DIR/edpcloud/src
as root:
[ Please replace the version numbers if needed ]cd $ED_BASE_DIR/edpcloud/src
tar xvf ed_fuse-2.9.5.tar.gz
cd fuse-2.9.5
You can install fuse under /usr/local or $ED_BASE_DIR/edpcloud
./configure –prefix=/usr/local
make
make install
Now add /usr/local/lib
/etc/ld.so.conf (assuming you installed fuse there)
modprobe fuse
ldconfig
Edit eddist.cfg replication configuration
We will need to create two links (also called replication sets). One going from svr1 to svr2 (l1) and one going from svr2 to svr1 (2). These will allow us to replicate two directions, thus called “bidirectional replication,” also known as “two-way sync.” File changes on svr1 will be synchronized with svr2 using the link (repset) l1. File changes on svr2 will be synchronized with svr1 using link l2.
Here is the content of eddist.cfg
<?xml version="1.0" encoding="UTF-8"?> <config name="enduradata" password="Addoud4d4ch1n1gh4T4s4" workers="4"> <link name="l1" password="foo"> <sender hostname="svr1" /> <receiver hostname="svr2" storepath="/" /> </link> <link name="l2" password="foo"> <sender hostname="svr2" /> <receiver hostname="svr1" storepath="/" /> </link> </config>
Notice that the data replication configuration above will not propagate deletes between the two systems. To propagate deletes, we need to add the keyword deletes=”1″ to eddist.cfg as the following example shows (But don’t do deletes=”1″ unless you know what you are doing).
<?xml version="1.0" encoding="UTF-8"?> <config name="enduradata" password="Addoud4d4ch1n1gh4T4s4" workers="4"> <link name="l1" password="foo"> <sender hostname="svr1" /> <receiver hostname="svr2" storepath="/" deletes="1"/> </link> <link name="l2" password="foo"> <sender hostname="svr2" /> <receiver hostname="svr1" storepath="/" deletes="1"/> </link> </config>
Now the directory /data will be mirrored between the two servers svr1 and svr2. Any changes made on svr1 are made on svr2, and any changes made on svr2 are made on svr1 using file synchronization. Please note that there is no distributed lock manager between the two servers, and the last writer wins (The same behavior as with NFS).
Start EDpCloud for Linux file replication services
Run the following command to start EDpCloud, and you will be on your way to synchronizing data between the two servers in real-time thus ensuring that data is backed up between the two.
. $ED_BASE_DIR/bin/edpcloud.sh startall
Run edstat to make sure edpcloud is running before you proceed to the next step. Do not proceed unless edpcloud is running without the real-time configuration. If you run into problems, check eddist.log to find out what is wrong with your data replication (Other logs can be found in $ED_BASE_DIR/logs)
Replicate some data by using:
edq -l linkname -r receivername -n dir_or_filename
run edstat again and verify that your files were replicated to the receiver side.
Do all the previous steps for all hosts in eddist.cfg. Once scheduled and on-demand replication and backup are working, proceed to configure the real time replication and synchronization.
Edit edfsmonitor.cfg, the Linux Real Time Replication Configuration
Add this line to the file and save it.
/data
This tells EDpcloud to monitor /data for changes and queue them for replication to the remote server.
Now that real time replication is configured, you need to start the real time file system changes, monitor:
Source in the environment (notice the dot):
cd /tmp
. $ED_BASE_DIR/bin/enduradata_env
edstartuwfs.sh
if all works, you should see the directories in edfsmonitor.cfg mounted (use df to see what is mounted)
Your real time bidirectional replication is now configured, and any files that change in directories, listed in edfsmonitor.cfg, on one system are replicated to the other remote system.
QUESTIONS?
The following is a list of related articles you may be interested in:
Share this Post