EnduraData story: why file replication and synchronization software?

Note: A reprint of an article from 2013 by Steve Olinger (SJO)

“For every complex problem there is a simple solution, and it is wrong” — attributed to H.L. Mencken

EDpCloud file replication topologies

– Why are we here? Why file synchronization and replication

The reasons for founding EnduraData were professional and personal – aligning with the desire to help solve complex and challenging problems – often based on human behavior. As many of us have discovered, there are no simple, practical solutions to complex problems.

Read on to find how EnduraData got into file synchronization and file replication.

Making complexity easier to manage

One reason for starting the company was to provide a better way for organizations to manage the massive amounts of data involved within a growing enterprise – enhance productivity, mitigate risk, lower total costs of ownership.

Context – a brief history of complex computing

In early computing, single users often worked on individual files (like documents or spreadsheets) in sequential order. How did we share these files? Mostly manually, using FTP (file transfer protocol), RCP (Remote Copy), Rdist (Remote distribution), FTP, and scp (secure copy). Then came databases, collaboration on files (like databases, graphics, text, video, etc.), and the need to access and distribute data to larger groups – within the network and across the enterprise (global? No problem). Today it’s the ever-increasing amounts of data creation with rapidly growing data devices and channels that drive this need. How do we manage this tsunami of data – turning it into something of value?

At EnduraData, we’re finding new (and we think better) ways to effectively share content between users, sites, business processes, enterprises, and stakeholders that span many geographic locations.

EDpCloud file replication

EDpCloud is a host-based replication solution that combines data distribution, remote online backup, and file synchronization. We streamline the process of moving data where you need it when you need it. Our solution works on Linux, Mac, Windows, Solaris, and many other UNIX systems meaning, if your enterprise has multiple operating systems, including those listed, you can share data seamlessly.
Data can be sent from one system to another, from many systems to a single system or multiple systems to multiple systems. Users can build simple or complex meshes(configured links) for disaster recovery, data distribution/aggregation, or workflow automation. The number of configured links that can be built within an enterprise is almost unlimited.

Relevant experience and expertise

A common theme at EnduraData is working to solve complex IT problems, creatively and effectively.
One example dates back to a team member’s experience working with the National Weather Service. This problem was how to accurately estimate water levels of the annual snowpack to predict how its melting would affect various geographies from the point of origin to points downriver.
Part of the solution involved creating a computer model of the geographic terrain in a specific area and designing an aerial survey program that used nuclear detectors to estimate snow water content, conducting snow depth sampling then running the program that calculated the amount of snow moisture that would melt and travel down the watershed and the rivers.

The computer processing requirements for this program were substantial. Even using existing supercomputers would require weeks of exclusive processing resources.

Our team member designed a process that partitioned the number and types of operations required into manageable tasks that could be processed by multiple computers (in a master/slave design) – using replication and load balancing.
The net result was a complex process that had required more than a week of supercomputer processing resources was split among several data centers and completing the computations in less than 24 hours, hence saving lives and property. In addition to saving calendar days of supercomputer time, human life, and personal and business property (and delivering the information faster), this model could be applied to other complex computation projects with the National Weather Service and the Military.

Another example comes from a team member’s critical development work at a company (before EnduraData) that served NASA.
NASA used data replication software to analyze film footage of the takeoff of space shuttle Discovery in July 2005. The film, known as the Ground Camera Ascent Imagery Project, was part of the stipulations of the Return to Flight program laid out by the Columbia Accident Investigation Board following the loss of the space shuttle Columbia on Feb. 1, 2003. Discovery’s mission in 2005 was the first human space flight undertaken by NASA post-Columbia disaster.

An investigation of Columbia’s demise had shown that a piece of foam had broken off the shuttle at launch and damaged one of the heat-resistant tiles on the orbiter, which caused the shuttle to break apart on re-entry into the Earth’s atmosphere. To try to prevent history from repeating itself, the high-definition film was taken from 26 different angles of Discovery’s launch, and then analyzed for any anomalies that could put the flight or the crew at risk.
NASA used the replication software to distribute film files and images of the Discovery launch to Kennedy Space Center in Fla., Johnson Space Center in Texas, and Marshall Space Flight Center in Ala.

The 26 camera feeds produced huge files; the smallest was 6 gigabytes in size. Moreover, they needed to go through a WAN and firewall built to NASA security standards. Finally, the product needed to fit into a tight budget — otherwise, NASA would have probably have gone with a hardware-based appliance, which proved too pricey.

The product was the only software-based replication product that met the many specialized aspects of NASA’s application: it fit the security requirements, worked with its Solaris operating system and met the minimum performance criterion, which was that 14 massive video files be replicated perfectly to the three locations within the first hour after Discovery’s launch.

Fast forward to 2020 and 2024, EnduraData is now being used to secure file replication for SSA.Gov , Medicare, Meidcaid and more.

EnduraData is formed

EnduraData was founded in 2006 to deliver improved tools for organizations that need a better way to share to distribute large amounts of data and eventually to protect it.

Problems our software helps solve

Data security
Data migration
Data movement and file sync
Workflow optimization
Data control and access
Business continuity and disaster recovery,
Data distribution
Client privacy
Regulatory compliance and the need to acquire more hardware
Software and personnel (IT spend).

Example

Problem: A growing company with multiple branches, faces the challenge of sharing data, including reports generated several times a day. Because the company made several acquisitions over time, several technologies make up its IT infrastructure (from various vendors like Microsoft, Sun Microsystems, Red Hat, IBM, and Apple). The company needs to have the data in a central location and to allow the various branches access to the same data.

Several issues compound the problem:

Multiple hardware and software systems vendors require additional solutions to deal with data distribution.
No one single vendor can deal with distributing data to/from all platforms used. If a disaster hits one or more branches, significant time and effort will be lost. Every day, the IT staff spends a substantial amount of time changing scripts and making manual changes to deal with data distribution.
The vendors’ solutions do not inter-operate with each other, making it necessary to purchase solutions from all vendors. Once in awhile, the network becomes congested or interrupted. Data may be delayed, or the system may need to retransmit it.

Therefore the company will have:

Significant time loss by IT staff
Islands and silos within the company that will not be able to access or share data
Significant duplication of efforts and costs (c) considerable management overhead
Redundant efforts to learn the solutions from multiple vendors and
Significant productivity issues stemming from data unavailability
Risk of non-compliance with Service Level Agreements or Regulations.

Using the EDpCloud cross-platform data distribution, the company will now be able to

1. Use a single solution to address the problems cited above since it is compatible between all platforms (Windows, Mac, Linux, and Solaris) from various vendors.
Windows machines will be able to distribute data to Linux and vice versa. The same is valid for all other platforms.
2. Consolidate data from all branches into a single location using controlled access. Distribute data from all departments and offices to all other branches and offices using controlled access.
4. Use one software solution to the job instead of using software from multiple vendors, with each one vendor solving the problem only for their platform.
5. The learning curve of using a single unified solution will not only solve the data distribution problem, but it will result in significant reductions in the total cost of ownership.
6. The company will reduce maintenance and management costs.
7. If the company needs access to source code to add new features or for further deep integration with the workflow, EnduraData is the only company that will license the source code.

Our mission

We solve IT and business problems. Our focus is to serve companies that move and process large amounts of data, storage, and data management.

We strive to build strong client relationships by treating clients as collaborative partners working towards mutual business success.
At the same time, we want to create a company that’s a rewarding place to work, where we provide meaningful projects – software development as an ongoing source of solutions.

Why work with us – how are we different (better)?

What makes our software unique is that we offer secure data distribution and file synchronization/replication and data backup/archiving that works on most hardware and most operating systems. Besides, we deliver a level of security that mitigates risk in multiple categories (i.e., client privacy, data loss, managing increasing amounts of data).
Our software solution automatically delivers secure data to where it is needed when it is required.

Click here to email us.