Data Engineering Pipelines for Real-Time AIML Decision-Making in Dynamic Business Environment
DOI:
https://doi.org/10.63282/3050-922X.ICRCEDA25-116Keywords:
Data Engineering, Real-Time Analytics, Artificial Intelligence, Machine Learning, Decision-Making, Dynamic Business Environments, Data Pipelines, Scalability, System IntegrationAbstract
In dynamic business environments, the ability to make real-time, data-driven decisions is crucial for maintaining a competitive edge. This paper explores the design and implementation of data engineering pipelines integrated with Artificial Intelligence (AI) and Machine Learning (ML) to facilitate real-time decision-making. We examine the architecture of such pipelines, focusing on data ingestion, processing, and analytics components that support AI/ML models. The study also addresses challenges including data latency, scalability, and system integration, offering solutions to optimize performance and reliability. Case studies from sectors like finance, healthcare, and manufacturing illustrate the practical applications and benefits of these integrated pipelines
References
[1] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 51(1), 107-113.
[2] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A Distributed Messaging System for Log Processing. Proceedings of the NetDB, 1-7.
[3] Marz, N., & Warren, J. (2015). Big Data: Principles and Best Practices of Scalable Realtime Data Systems. Manning Publications.
[4] Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). Spark: Cluster Computing with Working Sets. Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing.
[5] Ghoting, A., Krishnamurthy, A., & O’Callaghan, L. (2017). Data Engineering for Machine Learning: Challenges and Opportunities. IEEE Data Engineering Bulletin, 40(4), 43-53.
[6] Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794.
[7] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2015). Apache Flink: Stream and Batch Processing in a Single Engine. IEEE Data Engineering Bulletin, 38(4), 28-38.
[8] Kreps, J. (2014). Questioning the Lambda Architecture. O’Reilly Radar Blog. https://www.oreilly.com/radar/questioning-the-lambda-architecture/
[9] Chen, L., & Liu, Z. (2019). Real-Time Data Processing for Dynamic Business Decision-Making. Journal of Big Data, 6(1), 24.
[10] Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018). Data Management Challenges in Production Machine Learning. Proceedings of the 2017 ACM International Conference on Management of Data, 1723-1726.
[11] Liu, S., Lin, F., & Hu, X. (2020). Real-Time AI/ML-Driven Decision Systems in Healthcare: A Review. Artificial Intelligence in Medicine, 105, 101856.
[12] Nguyen, H., & Pathak, P. (2021). Scalable Feature Engineering for Real-Time Machine Learning Pipelines. Proceedings of the VLDB Endowment, 14(12), 2836-2849.
[13] Bhandari, S., Jain, A., & Verma, S. (2019). A Survey of Real-Time Analytics in IoT with AI/ML Integration. International Journal of Advanced Computer Science and Applications, 10(5), 276-285.
[14] Kim, H., Lee, J., & Choi, K. (2022). Stream Processing Architectures for Real-Time Decision Making: A Comparative Study. Journal of Systems and Software, 182, 111083.
[15] Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly Detection: A Survey. ACM Computing Surveys, 41(3), 1-58.
[16] Kirti Vasdev. (2025). “Enhancing Network Security with GeoAI and Real-Time Intrusion Detection”. International Journal on Science and Technology, 16(1), 1–8. https://doi.org/10.5281/zenodo.14802799
[17] Vegineni, Gopi Chand, and Bhagath Chandra Chowdari Marella. "Integrating AI-Powered Dashboards in State Government Programs for Real-Time Decision Support." AI-Enabled Sustainable Innovations in Education and Business, edited by Ali Sorayyaei Azar, et al., IGI Global, 2025, pp. 251-276. https://doi.org/10.4018/979-8-3373-3952-8.ch011
[18] Animesh Kumar, “Redefining Finance: The Influence of Artificial Intelligence (AI) and Machine Learning (ML)”, Transactions on Engineering and Computing Sciences, 12(4), 59-69. 2024.
[19] K. R. Kotte, L. Thammareddi, D. Kodi, V. R. Anumolu, A. K. K and S. Joshi, "Integration of Process Optimization and Automation: A Way to AI Powered Digital Transformation," 2025 First International Conference on Advances in Computer Science, Electrical, Electronics, and Communication Technologies (CE2CT), Bhimtal, Nainital, India, 2025, pp. 1133-1138, doi: 10.1109/CE2CT64011.2025.10939966.
[20] Gopichand Vemulapalli, Padmaja Pulivarthy, “Integrating Green Infrastructure With AI-Driven Dynamic Workload Optimization: Focus on Network and Chip Design,” in Integrating Blue-Green Infrastructure Into Urban Development, IGI Global, USA, pp. 397-422, 2025.
[21] Mohanarajesh, Kommineni (2024). Develop New Techniques for Ensuring Fairness in Artificial Intelligence and ML Models to Promote Ethical and Unbiased Decision-Making. International Journal of Innovations in Applied Sciences and Engineering 10 (1):47-59.
[22] Sahil Bucha, “Design And Implementation of An AI-Powered Shipping Tracking System For E-Commerce Platforms”, Journal of Critical Reviews, Vol 10, Issue 07, 2023, Pages. 588-596.
[23] Bhagath Chandra Chowdari Marella, “Scalable Generative AI Solutions for Boosting Organizational Productivity and Fraud Management”, International Journal of INTELLIGENT SYSTEMS AND APPLICATIONS IN ENGINEERING, vol. 11, no.10, pp. 1013–1023, 2023.
[24] Divya K, “Efficient CI/CD Strategies: Integrating Git with automated testing and deployment”, World Journal of Advanced Research and Reviews: an International ISSN Approved Journal, vol.20, no.2, pp. 1517-1530, 2023.
[25] Lakshmi Narasimha Raju Mudunuri, Venu Madhav Aragani. (2024). “Bill of Materials Management: Ensuring Production Efficiency”. International Journal of Intelligent Systems and Applications in Engineering, 12(23s), 1002–1012. Retrieved from https://ijisae.org/index.php/IJISAE/article/view/7102
[26] Mudunuri L.N.R.; (December, 2023); “AI-Driven Inventory Management: Never Run Out, Never Overstock”; International Journal of Advances in Engineering Research; Vol 26, Issue 6; 24-36
[27] Sudheer Panyaram, (2023), AI-Powered Framework for Operational Risk Management in the Digital Transformation of Smart Enterprises.
[28] Pulivarthy, P. (2024). Gen AI Impact on the Database Industry Innovations. International Journal of Advances in Engineering Research (IJAER), 28(III), 1–10.
[29] Praveen Kumar Maroju, "Assessing the Impact of AI and Virtual Reality on Strengthening Cybersecurity Resilience Through Data Techniques," Conference: 3rd International conference on Research in Multidisciplinary Studies Volume: 10, 2024. – 1
[30] Mohanarajesh, Kommineni (2024). Generative Models with Privacy Guarantees: Enhancing Data Utility while Minimizing Risk of Sensitive Data Exposure. International Journal of Intelligent Systems and Applications in Engineering 12 (23):1036-1044.
[31] RK Puvvada . “SAP S/4HANA Finance on Cloud: AI-Powered Deployment and Extensibility” - IJSAT-International Journal on Science and …16.1 2025 :1-14.
[32] DESIGNING OF SEPIC PFC BASED PLUG-IN ELECTRIC VEHICLE CHARGING STATION, Sree Lakshmi Vineetha Bitragunta, International Journal of Core Engineering & Management, Volume-7, Issue-01, 2022, PP-233-242.
