EDpCloud Real Time File System Change Monitor Configuration

Check for Updates of your file sync and replication software from www.enduradata.com

edfsmonitor.cfg

NAME

NAME

edfsmonitor.cfg - Windows and Linux Realtime File Replication Monitor Configuration

SYNOPSIS

EnduraData file replication monitor uses edfsmonitor.cfg as a configuration file for its real time file replication software.

The edfsmonitor.cfg is a text file that contains the path names to be monitored by the file system module for file replication and transfer. The real time file system replication monitor can monitor a large number of files and directories. The files and directories are queued for file sync, replication and transfer to the remote nodes. The file replication monitor captures the file changes in real time and passes the change to file replication service. It is important that you perform an initial file synchronization to populate the remote directory with existing content. This file synchronization can be performed any time before or after you start running edfsmonitor. This is required since edfsmonitor only tracks the new changes and not past changes.

edfsmonitor captures the changes to the file system and relays them to eddist replication service which journals the changes and queues them for data replication. edfsmonitor may relay the file changes to eddist at periodic times that can range from minutes after file changes changes. The lapse period may be configured using ED_Q_POLL_TIME environment variable but the system may override such parameters depending on available system resources.

If your performing bidirectional file replication, it is important that you know that if two writers on two different systems may overwrite each others changes (the last change that gets replicated wins because there is no distributed lock manager).

The following are some other important notes about the real time replication software:

Renaming directories is very expensive since a rename is a copy followed by a delete.

Always set the number of workers to 1 with real time replication and the system will optimize for the lost worker threads to guarantee data consistency.

If you want to create an exact mirror with replication, make sure you use incfile and exfile correctly (see eddist.cfg documentation) and to set deletes="1" because applications such as gzip create new .gz files and delete the old input files. Other applications may create temporary files that are deleted later. Unless you set deletes="1", temporary files will be replicated to the remote site but will not be deleted. As an alternative, add the temporary file name regular expressions to the the excludes list.

APPLYING THE CONFIGURATION

The file contains paths to be monitored. Each line must contain the absolute path (including the drive letter) to be monitored.

CONFIGURING THE FILE CHANGE PORT

Use the environment variable ED_REALTIME_PORT to setup the port on which the super server listens to file changes.

POLL TIME FOR FILE CHANGES

You can control how frequently that names of the files that changed are queued for replication by setting the ED_Q_POLL_TIME=nnn where nnn is the number of seconds where nnn is greater than zero. The shorter nnn is the more overhead you have on the system but the more frequent is the polling for changes. The larger is nnn the more memory is used to store the file names in memory. Use the default value until you fine tune ED_Q_POLL_TIME

Example 1 for windows

   c:\data
   c:\Users\john
   c:\Users\rime
   d:\incoming\documents
   d:\Documents

Example 2 for Linux

  /home
  /data

Please add individual file systems to replicate rather than adding the root (/) file system.

You can use the GUI to create or alter this file. You can also use a text editor such as notepad.

Once you edit the edfsmonitor.cfg, you may need to stop and start edfsmonitor service as follows for windows

        net stop edfsmonitor

        net start edfsmonitor

For Linux, you can use the following scripts to start and stop edfsmonitor.

        edstartuwfs.sh
        edstopuwfs.sh

Make sure no process or shell is using the mounted file system before you stop or start edfsmonitor.

Any errors encountered will be found in edfsmonitor.log or eduwfs.log located under the logs directory.

For Linux, extensive tracing can be set using $edpcloud/etc/jtrace file: Use J,K, ... trace flags.

Linux users must have util-linux version 2.18 or higher. The version of mount/unmout must support the "--fake" and "no-canonical options". Linux errors will be in eduwfs.log in the logs directory. You can download the util-linux from https://www.kernel.org/pub/linux/utils/util-linux

JOURNAL SERVER AND PORT

Normally edfsmonitor sends its journal to the localhost and eddist port. The receiving host can be changed by using the environment variable ED_FSMONITOR_JOURNAL_SERVER

IMPORTANT

Note 1: Once you configure your replication you and start edpcloud, make sure you perform the initial synchronization.

Note 2: If your server dies in the middle of synchronization or if you shutdown edpcloud, make sure your rerun edmfq to synchronize any data changes that may not have been journaled before the server or edpcloud stopped.

EDpCloud will not move files across file systems unless you export the environment variable ED_CROSS_FS_MV=1

SUPPORT

For more information contact <support@enduradata.com>

AUTHOR

A. A. El Haddi, elhaddi@ieee.org
A. Taouil taouil@enduradata.com