EDpCloud

Sync, Replicate, Protect Data

EDpCloud™: Find Files That Changed Before a Certain File Changed Or Seconds Before and Queue Them for Transfer.

• NAME
• SYNOPSIS
• DESCRIPTION
• PARAMETERS
• MORE EXAMPLES
• FILES
• SEE ALSO
• RETURN VALUE
• ENVIRONMENT
• SUPPORT
• AUTHOR

NAME

edmfq - Queue files that changed before now, before a file was modified, or post now or post a file modification

SYNOPSIS

edmfq [-l linkname] [-r receiver] [-s sourcepath ] [-p port ] [-n filetocompareto] [-q] [-t pretime] [-T posttime] [-x excludeexpression] B[-e excludefilelist] [-f filelist] [-m minimumfilesize] [-M maxfilesize] [-d]

DESCRIPTION

edmfq: The command line interface for queuing files that have changed while edpcloud was not running.

edmfq allows you to queue files or directories and their content for transfer and distribution to remote sites. This command analyses the file system and finds the files that have been modified pretime seconds before we started running or before the file filecompareto was modified. If posttime is specified then we also look forward posttime seconds and queue any files that were modified after the file filecompareto was modified.

This command is useful if you stop edpcloud or if the system halted without an appropriate shutdown. It allows you to re-synchronize data but starting only from the time when your system came backup going a few seconds back to a few minutes, days or months without resynchronzing the entire file system.

This command can be automatically called by edstartuwfs.sh in the case of Linux real time file replication. edmfq is faster than edq because it can filter files by a certain number of filters: i.e modification time, size, etc.

PARAMETERS

-C cksumfile

If cksumfile is specified then a csv file will be output with the list of files, their md5 checksum, modification time and size. Directory checksums will have "dir" instead of the md5.

-C can be combined with -t or -T to narrow the files to a certain range.

example:

edmfq -C /tmp/foo.log -s /tmp -t 10d

List only files under /tmp directory but list only the files modified in the last 10 days and store them in /tmp/foo.log. Add -d to include subdirectories as well. The command above will also generate the last modification times and the MD5 checksum.

edmfq -l link1 -s /home -t 3600 -q

Queue files modified in the last hour (60secx60min = 3600) under /home to link link1.

-d

By default edmfq will keep only files in the final reports, using -d will force it to consider a directory modified. Do not use -d if you are queing files to avoid doing the content of entire directories. Instead let the system do only files that changed and it will create directories as needed (unless you want to queue the entire directory regardless of modification times and in that case it is better to use edq instead of edmfq.

-l linkname

If no linkname is specified, we will queue data to all links.

-r receiver

If no receiver is specified, we will queue data to all receivers.

-p port

port where the file collector is running

If no port is specified we use the default specified in the system or in enduradata_env.

[-Z]

Use ssl certificates

-n filetocompareto

Compare time stamps on a candidate file with filetocompare to and use that as a reference. If no filetocompare to is specified we use the current time as a reference.

-m minimumfilesize

Do only files that are larger than minimumfilesize.

Example:

edmfq /tmp/foo.log -s /data/howto -m 80000 -q queue only files under /data/howto directory that have at least 80000 bytes in size.

-M maximumfilesize

Do only files that are larger less than Maximum file size.

Example: The following example will sync files from /data/howto if their size is less or equal to 4096 bytes edmfq /tmp/foo.log -s /data/howto -M 4096 -q

if -m and -M are combined then only files between the minumumsize and maximumsize will be synced.

This example will synchronize from /data/howto that have a file size between 2048 and 4096 bytes.

edmfq /tmp/foo.log -s /data/howto -m 2048 -M 4096 -q

-x excludeexpression

A set of regular expressions: All file names that match excludeexpression will not be synced.

-e expressionfilelist

This is the name of a file that contains one regular expression per line. File names that match the regular expressions will be excluded from replication.

-f filelist

filelist contains a list of files and directories to synchronize without regard to their modification times. The format is one file or directory name per line. Make sure no spaces or other characters exist before the start of the file name.

-s sourcepath

Name of directory to scan for files to sync.

-q

Queue all files that match our rules for replication. Files will not be synchronized unless -q is specified.

-t presec

Time in seconds to go back and find files that were modified presec seconds before now or before the filetocompare to (-n). If presec is zero then, we will use elapsed time of infinity. That will consider all files modified.

presec can be in the format nnnn[s|h|d|m|M|y" where s indicates that nnnn is in seconds, d in days, m in minutes, M in months, y in years.

-t 2d stands for the two days, 5M stands for 5 months, 15m stands for the last 15 minutes.

-T postsec

Look forward tseconds from when we started or when filetocompareto (-n) was modified and add the files to our rule matched.

MORE EXAMPLES

edmfq -t 3000 -T 30 -n /tmp/foo -s /p02/data -q

Find files in /p02/data that are 0 to 3000 seconds older than file /tmp/foo or that are at most 30 seconds newer than /tmp/foo and queue them for replication.

Same example as above but only sync files not larger than 520000 bytes:

edmfq -t 3000 -T 30 -n /tmp/foo -s /p02/data -q -M 520000

Same as above but this time synchronize files that are between 1000 bytes and 20000 bytes: edmfq -t 3000 -T 30 -n /tmp/foo -s /p02/data -q -m 1000 -M 20000

edmfq -t 300 -s /p02/data -q

Find files in /p02/data that changed between 0 and 300 seconds ago and queue them for replication.

To replicate files that have changed between now and two hours ago (2 * 60 * 60 seconds = 7200 seconds) under the path /p02/data:

edmfq -t 7200 -s /p02/data -q

To replicate files that have changed between now and 5 minutes before the last modification time of the file /home/zbob/dzproj/foo.c (5* 60 seconds = 300 seconds) under the path /home/zbob/dzproj:

edmfq -t 300 -s /home/zbob/dzproj -q

To automatically synchronize all files that have changed in the last 10 days (10*24*60*60 seconds) under /home

edmfq -t 864000 -s /home -q

To automatically synchronize all files that have changed in the last 30 days (10*24*60*60 seconds) under /home to link foo

edmfq -t 1m -s /home -q -l foo

or

edmfq -t 30d -s /home -q -l foo

Notice that if no -t fmodname option is given, the time is relative to the current time, otherwise we use the last modification time of the file specified with -s switch. If -T tesc is given , we also search tsec seconds after the last modification time of the file given by -s.

FILES

$ED_BASE_DIR/etc/edpasswd

SEE ALSO

edintro(8)

edresume(8)

edpause(8)

edjob(8)

eddist(8)

edstat(8)

edq(8)

eddist.cfg(5)

RETURN VALUE

The status is 0 on success.

ENVIRONMENT

ED_BASE_DIR: This is the name of the top directory where the file replication and filesync software is installed. This directory will also contain various configuration, journals and log files.

ED_COLLECTOR_PORT: This environment variable is set to the port on which EnduraData eddist server listens to management commands. Set this environment variable to the port on which to listen for commands.

ED_DISTRIBUTION_PORT: This environment variable is set to the port on which EnduraData eddist server listens to data synchronization services. Set this environment variable to the port on which to listen for data distribution connections.

$ED_BASE_DIR/bin/enduradata_env has all the environment variables we depend on

SUPPORT

For more information contact <support@enduradata.com>

AUTHOR

A. A. El Haddi, elhaddi@ieee.org