Volume IV-4 | Copyright
ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., IV-4, 243-246, 2018
https://doi.org/10.5194/isprs-annals-IV-4-243-2018
© Author(s) 2018. This work is distributed under
the Creative Commons Attribution 4.0 License.

  19 Sep 2018

19 Sep 2018

NMSTREAM: A SCALABLE EVENT-DRIVEN ETL FRAMEWORK FOR PROCESSING HETEROGENEOUS STREAMING DATA

F. Xiao, C. Li, Z. Wu, and Y. Wu F. Xiao et al.
  • Chinese Academy of Surveying and Mapping, Beijing, China

Keywords: Streaming data, Extract-Transform-Load, Apache Flume, Apache Cassandra

Abstract. ETL (Extraction-Transform-Load) tools, traditionally developed to operate offline on historical data for feeding Data-warehouses need to be enhanced to deal with continuously increased streaming data and be executed at network level during data streams acquisition. In this paper, a scalable and web-based ETL system called NMStream was presented. NMStream is based on event-driven architecture and designed for integrating distributed and heterogeneous streaming data by integrating the Apache Flume and Cassandra DB system, and the ETL processes were conducted through the Flume agent object. NMStream can be used for feeding traditional/real-time data-warehouses or data analytic tools in a stable and effective manner.

Download & links