DataOps and Agile Data Engineering: Accelerating Data-Driven Decision-Making
DOI:
https://doi.org/10.63282/3050-922X.IJERET-V1I1P104Keywords:
DataOps, Agile Data Engineering, Data Governance, Data Pipelines, DevOps, Big DataAbstract
Business intelligence, translated as data processing and analysis results for company decision-making, is integral to the modern business environment. DataOps and Agile Data Engineering cover a model concerning the automation and making of strong data workflows reaching maximum reliability and scalability. This paper focuses on using DataOps in data engineering and how it aids in integrating agile methodologies to aid decision-making. Some of the major factors which need to be addressed include: data management, data quality and pipeline management challenges. This paper explains how DataOps builds up the idea of data engineering using comparison with DevOps in detail. In addition, there is a case of practising real-time analytics that provides insight into how the usage of these methodologies occurs. In the last section, the authors have pointed out the set of recommendations, possible future trends in using DataOps, and its possibilities to bring transformation in a world where data are increasingly becoming important
References
[1] Marr, B. (2017). Data strategy: How to profit from big data, analytics and the Internet of things. Kogan Page Publishers.
[2] Humble, J., & Farley, D. (2010). Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education.
[3] Kim, G., Humble, J., Debois, P., Willis, J., & Forsgren, N. (2021). The DevOps handbook: How to create world-class agility, reliability, & security in technology organisations. It Revolution.
[4] Gorelik, A. (2019). The enterprise big data lake: Delivering the promise of big data and data science. O'Reilly Media.
[5] Atwal, H. (2020). Practical DataOps. Practical DataOps (1st ed.). Apress Berkeley, CA. https://doi. org/10.1007/978-1-4842-5104-1.
[6] Bussa, S., & Hegde, E. (2024). Evolution of Data Engineering in Modern Software Development. Journal of Sustainable Solutions, 1(4), 116-130.
[7] Brady, S. (2015). Agile and Incremental Software Development in the Defense Acquisition System. In Annual NDIA System Engineering Conference (pp. 1-27).
[8] Grady, N. W., Payne, J. A., & Parker, H. (2017, December). Agile big data analytics: AnalyticsOps for data science. In 2017 IEEE international conference on big data (big data) (pp. 2331-2339). IEEE.
[9] Fawzy, A., Tahir, A., Galster, M., & Liang, P. (2025). Exploring data management challenges and solutions in agile software development: a literature review and practitioner survey. Empirical Software Engineering, 30(3), 1-61.
[10] Kennedy, O. A., Ahmad, Y., & Koch, C. (2011). Agile views in a dynamic data management system. In CIDR 2011, Fifth Biennial Conference on Innovative Data Systems Research (pp. 284-295).
[11] Vestues, K., Hanssen, G. K., Mikalsen, M., Buan, T. A., & Conboy, K. (2022, June). Agile data management in NAV: a case study. In International Conference on Agile Software Development (pp. 220-235). Cham: Springer International Publishing.
[12] Alagar, DataOps: Bridging the Gap Between Data Engineering and Data Science, IABAC, 2023. online. https://iabac.org/blog/dataops-bridging-the-gap-between-data-engineering-and-data-science#:~
[13] Lillie, T., & Eybers, S. (2019). Identifying the constructs and agile capabilities of data governance and data management: A literature review. In Locally Relevant ICT Research: 10th International Development Informatics Association Conference, IDIA 2018, Tshwane, South Africa, August 23-24, 2018, Revised Selected Papers 10 (pp. 313-326). Springer International Publishing.
[14] Haidabrus, B., Grabis, J., & Protsenko, S. (2021). Agile project management based on data analysis for information management systems. Design, simulation, manufacturing: the innovation exchange, 174-182.
[15] Nasser, T., & Tariq, R. S. (2015). Big data challenges. J Comput Eng Inf Technol 4: 3. doi: http://dx. Doi. Org/10.4172/2324, 9307(2).
[16] Mainali, K., Ehrlinger, L., Matskin, M., & Himmelbauer, J. (2021). Discovering DataOps: a comprehensive review of definitions, use cases, and tools. In DATA ANALYTICS 2021 The Tenth International Conference on Data Analytics.
[17] Tim Mucci, Cole Stryker, Mark Scapicchio, What is DataOps?, IBM, online. https://www.ibm.com/think/topics/dataops
[18] Jan, B., Farman, H., Khan, M., Imran, M., Islam, I. U., Ahmad, A., ... & Jeon, G. (2019). Deep learning in big data analytics: a comparative study. Computers & Electrical Engineering, 75, 275-287.
[19] Hammami, J., & Khemaja, M. (2019). Towards agile and gamified flipped learning design models: Application to the system and data integration course. Procedia Computer Science, 164, 239-244.
[20] Aitken, A., & Ilango, V. (2013, January). A comparative analysis of traditional software engineering and agile software development. In 2013 46th Hawaii International Conference on System Sciences (pp. 4751-4760). IEEE.
[21] Song, H., Liu, X., & Song, M. (2023). Comparative study of data-driven and model-driven approaches in prediction of nuclear power plants operating parameters. Applied Energy, 341, 121077.
