Introduction to EDpCloud file sync with pre and post processing directives
Many business processes need to extract data, to transform it and to process it before sending it to a remote location to be transformed further or to be loaded into other databases or fed to other processes downstream. Data can be sent to the remote locating using (File Transfer Protocol) FTP, HTTP or other synchronization means. This article discusses how to use EDpCloud filesync software with pre and post processing capabilities to accomplish such a task automatically.
Pre-processing with file sync software (pre)
Figure 1 illustrates how EDpCloud post and pre processing can be used to automate data extraction, transformation, loading and report distribution to many locations and in sort creating a sort of ETL that deals with 24/7 available slot ceiling issues.
EDpCloud can be configured to call an external script or program to accomplish tasks before files are synchronized with one or more remote locations. The data synchronization program is handed a group of files to synchronize. Before synchronization takes place, if a pre sync directive is given, the sender will run the script specified in the “pre” directive to process the files or execute other tasks. The next step, synchronization starts (as shown in the figure below). When the receiving end gets a request to synchronize data, it also can call a script if one is specified in the “pre” directive for the receiving end. Again, the pre
synchronization script is handed a list of files that will be pre-processed. For example the receiving end may prefer to delete some files, rename them or take whatever action is needed before synchronization takes place.
Post processing with EDpCloud file sync software(post)
The “post” processing directive is specified in the configuration by setting post to the name of the script or of the program that will be called for each group of files that were synchronized (along with the status of the synchronization). This is done on both the receiver and the sender end.
Examples of uses
Figure 1 depicts a scenario where a process extracts some data from one more more databases or files, generates some reports, saves the output in one or multiple files then sends the files or the folders to be synced with one or more locations. Going beyond simple file and folder sync, once the reports arrive on the remote location, the data is ingested into a database and a script is run to generate more reports and distribute the new data to one or more new locations.
You can build complex, dynamic and automatic extract transform and distribute configurations using pre and post directives that are part of EDpCloud file sync software.
Learn about additional features of EDpCloud file replication and file sync software.
https://www.enduradata.com/data-protection-mac-windows-linux-solaris-aix/
https://www.enduradata.com/combining-file-transfer-file-archiving/
https://www.enduradata.com/file-replication-software-a-cross-platform-solution/
https://www.enduradata.com/automatic-backup-and-file-sync-for-windows-mac-linux-solaris-aix-and-unix/
https://www.enduradata.com/file-transfer-file-replication-papers//
Share this Post