IRMA-International.org: Creator of Knowledge
Information Resources Management Association
Advancing the Concepts & Practices of Information Resources Management in Modern Organizations

A Novel Spatial Data Pipeline for Orchestrating Apache NiFi/MiNiFi

A Novel Spatial Data Pipeline for Orchestrating Apache NiFi/MiNiFi
View Sample PDF
Author(s): Chase D. Carthen (University of Nevada, Reno, USA), Araam Zaremehrjardi (University of Nevada, Reno, USA), Vinh Le (University of Nevada, Reno, USA), Carlos Cardillo (University of Nevada, Reno, USA), Scotty Strachan (Nevada System of Higher Education, USA), Alireza Tavakkoli (University of Nevada, Reno, USA), Frederick C. Harris Jr. (University of Nevada, Reno, USA)and Sergiu M. Dascalu (University of Nevada, Reno, USA)
Copyright: 2024
Volume: 12
Issue: 1
Pages: 14
Source title: International Journal of Software Innovation (IJSI)
Editor(s)-in-Chief: Roger Y. Lee (Central Michigan University, USA)and Lawrence Chung (The University of Texas at Dallas, USA)
DOI: 10.4018/IJSI.333164

Purchase

View A Novel Spatial Data Pipeline for Orchestrating Apache NiFi/MiNiFi on the publisher's website for pricing and purchasing information.

Abstract

In many smart city projects, a common choice to capture spatial information is the inclusion of lidar data, but this decision will often invoke severe growing pains within the existing infrastructure. In this article, the authors introduce a data pipeline that orchestrates Apache NiFi (NiFi), Apache MiNiFi (MiNiFi), and several other tools as an automated solution to relay and archive lidar data captured by deployed edge devices. The lidar sensors utilized within this workflow are Velodyne Ultra Puck sensors that produce 6-7 GB packet capture (PCAP) files per hour. By both compressing the file after capturing it and compressing the file in real-time; it was discovered that GZIP and XZ both saved considerable file size being from 2-5 GB, 5 minutes in transmission time, and considerable CPU time. To evaluate the capabilities of the system design, the features of this data pipeline were compared against existing third-party services, Globus and RSync.

Related Content

. © 2024.
Yogesh M. Kamble, Raj B. Kulkarni. © 2024. 10 pages.
Zachary Estreito, Vinh Le, Frederick C. Harris Jr., Sergiu M. Dascalu. © 2024. 15 pages.
Chase D. Carthen, Araam Zaremehrjardi, Vinh Le, Carlos Cardillo, Scotty Strachan, Alireza Tavakkoli, Frederick C. Harris Jr., Sergiu M. Dascalu. © 2024. 14 pages.
Partha Ghosh, Takaaki Goto, Leena Jana Ghosh, Giridhar Maji, Soumya Sen. © 2024. 15 pages.
Megha Bhushan, Utkarsh Verma, Chetna Garg, Arun Negi. © 2024. 14 pages.
Deepkiran Munjal, Laxman Singh, Mrinal Pandey, Sachin Lakra. © 2023. 25 pages.
Body Bottom