Data agility through clustered edge computing and stream processing

Abstract

© 2018 John Wiley & Sons, Ltd. The Internet of Things is underpinned by the global penetration of network-connected smart devices continuously generating extreme amounts of raw data to be processed in a timely manner. Supported by Cloud and Fog/Edge infrastructures – on the one hand, and Big Data processing techniques – on the other, existing approaches, however, primarily adopt a vertical offloading model that is heavily dependent on the underlying network bandwidth. That is, (constrained) network communication remains the main limitation to achieve truly agile IoT data management and processing. This paper aims to bridge this gap by defining Clustered Edge Computing – a new approach to enable rapid data processing at the very edge of the IoT network by clustering edge devices into fully functional decentralized ensembles, capable of workload distribution and balancing to accomplish relatively complex computational tasks. This paper also proposes ECStream Processing that implements Clustered Edge Computing using Stream Processing techniques to enable dynamic in-memory computation close to the data source. By spreading the workload among a cluster of collocated edge devices to process data in parallel, the proposed approach aims to improve performance, thereby supporting agile data management. The experimental results confirm that such a distributed in-memory approach to data processing at the very edge of an IoT network can outperform currently adopted Cloud-enabled architectures, and has the potential to address a wide range of IoT-related data-intensive time-critical scenarios.