EDpCloud
Sync, Replicate, Protect Data

EDpCloud™: EDpCloud with rclone or other sync.












Using other transport such as rclone sync with EDpCloud


Check for Updates of your file sync and replication software from www.enduradata.com

ed_senderprovider



NAME

ed_senderprovider - Replicating to other clouds without spinning a virtual machine using third party transport.

SYNOPSIS

EDpCloud can be used to sync data between VMs, VMs and physical machines, physical servers and VMs and other physical machines. However, this document will describe how you can sync between VMs, Physical machines and various cloud providers such as google cloud platform, Amazon S3, Wasabi, etc without spinning a VM on the cloud provider.

ed_senderprovider can be used to sync data using third party protocols. The default is rclone but you can tailor ed_senderprovider to use other protocols. Some users have special workflows that depend on third party or need to use other encryptions, etc. EDpCloud uses ed_senderprovider as a glue to such workflows. Most users use EDpCloud transport layer but this document will allow users to extend EDpCloud with their own transport layer.

Parameters to pass to ed_senderprovider

Where:

Where:

--

   11|     0|   0|               17360|                   0|0|/home/eh/sandbox/trunk/tamda/pkg/LINUX/x64/debug/edcdist/bin/ed_license_info

   12|     0|   0|             1176848|                   0|0|/home/eh/sandbox/trunk/tamda/pkg/LINUX/x64/debug/edcdist/bin/ed_is_license_valid

   13|     0|   0|               14489|               14489|0|/home/eh/sandbox/trunk/tamda/src/edpwebgui/unix/Makefile.unix

--

  90: Modified file

  5120: Renamed File

  2048: Delete file or directory

  -2048: Delete file or directory and override permissions

  110004:Symbolic link

Other operations are supported by EDpCloud but are not used by the ed_senderprovider.

--

90|/home/eh/presentations/fsync.ppt|

2048|/home/eh/presentations/bars.ppt|

5120|/home/eh/presentations/newbar.ppt|/home/eh/presentattions/oldbar.ppt

Configurations for using rclone

1. Install edpcloud and configure edpcloud
2. Install rclone or whatever transport layer you want to use.

In order to use a third party cloud platform with rclone for example you need to change two configurations as follows:

rctransport.conf for rclone

Example of rctransport.conf when using rclone

--

        [WASABI]
        type = s3
        provider = Wasabi
        env_auth = false
        access_key_id = KEYID028ExampleOnly2
        secret_access_key = FoobarKeyHEREforYouToSeeButGetyourOWNKey
        endpoint = s3.eu-central-1.wasabisys.com
        acl = private
        
        [AWSS3]
        type = s3
        provider = AWS
        env_auth = false
        access_key_id = GETYOUROWNACCESSID18
        secret_access_key = GetYourownSecretKeyBecauseThisisExample3
        region = us-west-2
        location_constraint = s3.us-west-2.amazonaws.com
        acl = private
        server_side_encryption = AES256
        
        
        # OK when using fine grained permissions
        [gcs]
        type = google cloud storage
        scope = drive
        service_account_file = /usr/local/enduradata/google-cloud/project1-name-757312-813b361a9097.json
        endpoint = https://storage.googleapis.com
        object_acl = bucketOwnerFullControl
        bucket_acl = bucketOwnerFullControl
        
        # OK only when using non fine grained permissions (default bucket)
        [gd1]
        type = google cloud storage
        scope = drive
        client_id = 999999d99d9d999999999
        service_account_file = /usr/local/enduradata/google-cloud/project1-name-757312-813b361a9097.json
        bucket_policy_only = true
        
        # Sync to Google drive
        #https://rclone.org/drive/
        #https://rclone.org/drive/#making-your-own-client-id
        [gdrive2]
        type = drive
        client_id = 29ab28d78ex8-get-your-own-key-seeabove-linksl.apps.googleusercontent.com
        client_secret = yoursecretkey-here-getit
        scope = drive
        token = {"access_token":"your-access-token-here-see-link-above-tosync-to-google-drive-more-token-info-here-must-use-the-docs-above-to-het-thiskey-andndnkdkdkqkkdkkVakkdkAKDKa_dfkk9sJoul-tested100","token_type":"Bearer","refresh_token":"1//0fBWhq7zxL3MLCgYIARAAGA8SNwF-L9IryETaVpK66P3P-RAUHfzP1hJfuM-yMIYUv3Vox4SRO2fbZVR8iHFSWcZTNKbMiEhFPwQ","expiry":"2010-04-12T11:41:14.7e9e9999e-06:00"}
        team_drive =

EDpCloud eddist.cfg configuration

The manual page for eddist.cfg describes how to configure EDpCloud.

The following is an example of an eddist.cfg configuration when using a third party transport layer such as rclone.
 
	<?xml version="1.0" encoding="UTF-8"?>
	        <config name="Network_Configuration" rowsize="8192">

          <link name="googledrive" isrealtime="1" workers="4" deletes="1" forcedelete="1" >

                <sender hostname="localhost" alias="*" password="foo" incfile="/usr/local/enduradata/edpcloud/etc/googledrive_includes"/>

                <receiver hostname="orion"   cloudprovider="googledrive" storepath="enduradatabucket1/" password="foo"/>

           </link>

        </config>

ed_senderprovider can be renamed and called to do other processing as shown below:

--

        1. Rename ed_senderprovider to ed_senderprovider.org

        2. Create a script or binary called ed_senderprovider

        3. Do other work before calling ed_senderprovider.org

        4. Call ed_senderprovider.org to sync

        5. Do other work 

        6. Create the status file and exit with status or non zero for failures

FILES

$ED_BASE_DIR/etc/eddist.cfg

$ED_BASE_DIR/etc/edpasswd

$ED_BASE_DIR/etc/debugcloud when present, ed_senderprovider prints more debug output.

$ED_BASE_DIR/etc/rc.params A one line list of additional parameters to pass to ed_senderprovider. This can also be set using the keyword cloudparams in eddist.cfg.

$ED_BASE_DIR/etc/rctransport.conf

This is the default configuration file of the transport. It can be overriden by the environment variable ED_PROVIDER_CFG

SEE ALSO

edintro(8)

edresume(8)

edpause(8)

edjob(8)

eddist(8)

edstat(8)

edmfq(8)

eddist.cfg(5)

RETURN VALUE

The status is 0 on success.

ENVIRONMENT

ED_BASE_DIR

This is the name of the top directory where the file replication and filesync software is installed. This directory will also contain various configuration, journals and log files.

ED_PROVIDER_CFG

Name of configuration file for third party transport. The default is $ED_BASE_DIR/etc/rctransport.conf

SUPPORT

For more information contact <support@enduradata.com>

AUTHOR

A. A. El Haddi, elhaddi@ieee.org