Activate Unlimited IoT Data with WANdisco Edge to Cloud

By Tony Velcich
May 04, 2022

The exponential growth of data is well-understood and widespread and is being driven in large part by Internet of Things (IoT) data. Examples of this data include information generated from industrial sensors, personal devices, and unstructured data sources such as video, audio, radar, and lidar (light detection and ranging). These types of data and others continue to grow in size and importance.

Access to more and larger amounts of data has been unexpectedly valuable. The information derived from data is the fundamental driver of modern business and scientific advancements that have previously seemed fanciful or impossible — examples of which include mundane but important supply chain and maintenance optimizations; the life-saving potential of self-driving cars that exceed the performance of human drivers; or the ability to produce images of black holes that were previously considered impossible to obtain. There are few human endeavors that are unaffected by this explosion of information. Equally important are related systems that scale with increasing demands for capture, storage, and processing of data. And the cloud is at the center of meeting these demands.

The combination of the scale, agility, and capacity of cloud computing, and the surprising effectiveness of machine learning (ML) techniques that emerge at scale, have, therefore, become drivers themselves for further advances. Cloud service providers such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) are innovating at a pace that far exceeds the ability of organizations that were previously lauded as technology innovators. Such innovation produces benefits for every business because the resulting capabilities are available on demand to anyone on an operational cost model. You can use the same infrastructure that supports the world’s most sophisticated and largest-scale computing environments, and do so without needing to commit beyond your known needs.

Activate enterprise-scale IoT datasets to fuel cloud analytics, AI, and ML

The ability to unleash data trapped and siloed in legacy on-premises systems and rapidly monetize it at scale is what WANdisco calls “data activation.” Data activation requires the combination of data collection, filtering, aggregation, and storage at the edge (IoT), and then moving that data to the cloud, where it can be stored inexpensively and durably, processed at any scale, and accessed from the wealth of innovative ML and AI services that are available in the cloud.

Edge computing, or IoT, data sets are unique in terms of data volumes and distribution. Edge environments by definition are distributed and typically have no single central point of data aggregation. While low-volume data ingest can be streamed directly to the cloud, large-scale data generation requires filtering and aggregation at the edge, resulting in data sets that are typically held in staging locations. IoT data sets can also be accumulated continuously or in batches. Sensor networks, device telemetry, video and audio recordings, and other types of time-series information will be created over time by potentially large numbers of generators.

This type of data landscape can make it challenging for chief data officers and other data leaders to wrangle and activate their company’s valuable data for business impact. Some of the barriers that stand in the way of digital success include data accruing faster than companies can handle, data being too distributed, data taking too long to get to the cloud and therefore too long to analyze, and disruption moving data to the cloud.

According to findings from a recent Wakefield Research report, more than two-thirds (69%) of the data leaders polled said that the inability to extract value from data — fueled by the barriers mentioned above — is holding back their company’s digital transformation.

Data leaders may find themselves at different stages of their company’s edge-to-cloud journeys. Despite which phase of the journey you are in, the right data activation strategy can help position your organization for success, now and in the future.

Wandisco Edge to Cloud: a breakthrough approach to data activation

WANdisco Edge to Cloud addresses the complexity and challenges companies face in moving large volumes of IoT data from the edge to the cloud. Edge to Cloud takes advantage of WANdisco’s unique capabilities to manage the migration of data sets at scale, along with our deep integration with the unique facilities of all the major public cloud service providers. WANdisco Edge to Cloud is designed to provide the most efficient, scalable, and manageable approach to making the massive volumes of IoT data available in your preferred cloud environment.

Edge to Cloud not only can be used for small data cloud transfers, but also for transferring petabytes and exabytes. But many of the features it provides are also applicable to smaller data transfers, with clear benefits for the management, visibility, control, and automation for data transfer.

WANdisco Edge to Cloud is designed to provide the most efficient, scalable, and manageable approach to making the massive volumes of IoT data available in your preferred cloud environment.

Using WANdisco Edge to Cloud is simple: First, you deploy one or many instances of the agent to edge locations that have access to your data, which can be in raw form or are filtered or processed in files typically located in staged storage at the edge. The agent provides the intelligence at the edge to initiate the transfer of data when required, from what can be a distributed edge infrastructure.

Second, you manage the collection of agents through the WANdisco user interface, command-line tool, or a REST API, giving you a variety of ways to coordinate and orchestrate the activities of the migrator agents.

Third, you initiate data movement and control its use of limited resources, such as bandwidth, to your choice of cloud target, which can include AWS, Azure, GCP, and other cloud providers. Last, you can monitor the progress and pace of data movement, ensuring that you can achieve the targets required for making data available in the cloud (see Figure 1).

Figure 1. The WANdisco user interface is used to manage and monitor data movement jobs.

Data transfer using WANdisco Edge to Cloud can be performed once, repeated when required, or orchestrated using external triggers and tooling. Operating a scalable, distributed transfer platform enables you to cope with arbitrarily large datasets and activate those datasets for your ML and AI operations at scale in the cloud — easily, securely, and with full visibility into this critical aspect of data activation.

Learn More

---