Building High-Performance ETL Pipelines with Incremental Data Loading
DOI:
https://doi.org/10.63282/3050-922X.IJERET-V6I1P107Keywords:
ETL, Data Integration, Load optimization, CDC, Checksum, Incremental LoadAbstract
ETL (Extract, Transform, Load) pipelines play a critical role in modern data processing and analytics. Traditional full data loads present challenges such as performance bottlenecks, increased resource consumption, and inefficiency. Incremental data loading emerges as a robust solution, enabling optimized processing by handling only new or changed data. This piece delves into the basics of incremental data loading, benefits, major strategies, best practices, and technologies underpinning high-performance ETL pipelines
References
[1] Biswas, Neepa, Anamitra Sarkar, and Kartick Chandra Mondal. "Efficient incremental loading in ETL processing for real-time data integration." Innovations in Systems and Software Engineering 16, no. 1 (2020): 53-61.
[2] Dhamotharan Seenivasan, "ETL (Extract, Transform, Load) Best Practices," International Journal of Computer Trends and Technology, vol. 71, no. 1, pp. 40-44, 2023. Crossref, https://doi.org/10.14445/22312803/IJCTT-V71I1P106
[3] Rahman, Nayem. "Incremental load in a data warehousing environment." International Journal of Intelligent Information Technologies (IJIIT) 6, no. 3 (2010): 1-16.
[4] Wiener, J., and J. Naughton. Incremental loading of object databases. Stanford InfoLab, 1996.
[5] Jörg, Thomas, and Stefan Dessloch. "Formalizing ETL jobs for incremental loading of data warehouses." Datenbanksysteme in Business, Technologie und Web (BTW)–13. Fachtagung des GI-Fachbereichs" Datenbanken und Informationssysteme"(DBIS) (2009).
[6] Dhamotharan Seenivasan, "Improving the Performance of the ETL Jobs," International Journal of Computer Trends and Technology, vol. 71, no. 3, pp. 27-33, 2023. Crossref, https://doi.org/10.14445/22312803/IJCTT-V71I3P105
[7] Mekterović, Igor, and Ljiljana Brkić. "Delta view generation for incremental loading of large dimensions in a data warehouse." In 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1417-1422. IEEE, 2015.
[8] Julakanti, Sivananda Reddy, Naga Satya Kiranmayee Sattiraju, and Rajeswari Julakanti. "Incremental Load and Dedup Techniques in Hadoop Data Warehouses." NeuroQuantology 20, no. 5 (2022): 5626-5636.
[9] Hsieh, Hui-Ching, and Mao-Lun Chiang. "The incremental load balance cloud algorithm by using dynamic data deployment." Journal of Grid Computing 17 (2019): 553-575.
[10] Henke, Elisa, Yuan Peng, Ines Reinecke, Michéle Zoch, Martin Sedlmayr, and Franziska Bathelt. "An extract-transform-load process design for the Incremental Loading of German Real-World Data based on FHIR and OMOP CDM: Algorithm Development and Validation." JMIR Medical Informatics 11 (2023): e47310.
[11] Murray, Derek G., Frank McSherry, Michael Isard, Rebecca Isaacs, Paul Barham, and Martin Abadi. "Incremental, iterative data processing with timely dataflow." Communications of the ACM 59, no. 10 (2016): 75-83.
[12] Dhamotharan Seenivasan, Muthukumaran Vaithianathan, 2023. "Real-Time Adaptation: Change Data Capture in Modern Computer Architecture" ESP International Journal of Advancements in Computational Technology (ESP-IJACT) Volume 1, Issue 2: 49-61, https://www.espjournals.org/IJACT/ijact-v1i2p106
[13] Jagadish, H. V., P. P. S. Narayan, Sridhar Seshadri, S. Sudarshan, and Rama Kanneganti. "Incremental organization for data recording and warehousing." In VLDB, pp. 16-25. ResearchGate GmbH, 1997.
[14] Vijayalakshmi, M., and R. I. Minu. "Incremental load processing on ETL system through cloud." In 2022 International Conference for Advancement in Technology (ICONAT), pp. 1-4. IEEE, 2022.
[15] Zhang, Xufeng, Weiwei Sun, Wei Wang, Yahui Feng, and Baile Shi. "Generating incremental ETL processes automatically." In First International Multi-Symposiums on Computer and Computational Sciences (IMSCCS'06), vol. 2, pp. 516-521. IEEE, 2006.
[16] Ong, Toan C., Michael G. Kahn, Bethany M. Kwan, Traci Yamashita, Elias Brandt, Patrick Hosokawa, Chris Uhrich, and Lisa M. Schilling. "Dynamic-ETL: a hybrid approach for health data extraction, transformation and loading." BMC medical informatics and decision making 17 (2017): 1-12.
[17] Behrend, Andreas, and Thomas Jörg. "Optimized incremental ETL jobs for maintaining data warehouses." In Proceedings of the Fourteenth International Database Engineering & Applications Symposium, pp. 216-224. 2010.
[18] Reddy, V. Mallikarjuna, and Sanjay K. Jena. "Active datawarehouse loading by tool based ETL procedure." (2010).
[19] Dhamotharan Seenivasan, "Critical Security Enhancements for ETL Workflows:Addressing Emerging Threats and Ensuring Data Integrity",International Journal of Innovative Research in Computer and Communication Engineering(IJIRCCE), Volume 12, Issue 3, March 2024, pp. 1301-1313, https://ijircce.com/admin/main/storage/app/pdf/XGaYnZXucV5i5bL5exiKjJYN9DX7xm7B0GTq8ivj.pdf
[20] Bathani, Ronakkumar. "Optimizing Etl Pipelines for Scalable Data Lakes in Healthcare Analytics." International Journal on Recent and Innovation Trends in Computing and Communication 9, no. 10 (2021): 17-24.
[21] Atmaja, I. Putu Medagia, Ari Saptawijaya, and Siti Aminah. "Implementation of change data capture in ETL process for data warehouse using HDFS and Apache spark." In 2017 International Workshop on Big Data and Information Security (IWBIS), pp. 49-55. IEEE, 2017.
[22] Suleykin, Alexander, and Peter Panfilov. "Metadata-driven industrial-grade ETL system." In 2020 IEEE International Conference on Big Data (Big Data), pp. 2433-2442. IEEE, 2020.
[23] Khan, Bilal, Saifullah Jan, Wahab Khan, and Muhammad Imran Chughtai. "An Overview of ETL Techniques, Tools, Processes and Evaluations in Data Warehousing." Journal on Big Data 6 (2024).
[24] Arsyad, Zulkifli. "Analisis Dynamic ETL Incremental Load untuk Data Integration Datawarehouse." INTERNAL (Information System Journal) 4, no. 2 (2021): 102-112.
[25] Dhamotharan Seenivasan, "Effective Strategies for Managing Slowly Changing Dimensions in Data Warehousing", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and ISSN Approved), ISSN:2349-5162, Vol.9, Issue 4, page no. ppi492-i496, April-2022,http://www.jetir.org/papers/JETIR2204861.pdf
[26] Gour, Vishal, S. S. Sarangdevot, Govind Singh Tanwar, and Anand Sharma. "Improve performance of extract, transform and load (ETL) in data warehouse." Int. Journal on Comp. Sci. and Eng 2, no. 3 (2010): 786-789.
[27] Qu, Weiping, Vinanthi Basavaraj, Sahana Shankar, and Stefan Dessloch. "Real-time snapshot maintenance with incremental ETL pipelines in data warehouses." In Big Data Analytics and Knowledge Discovery: 17th International Conference, DaWaK 2015, Valencia, Spain, September 1-4, 2015, Proceedings 17, pp. 217-228. Springer International Publishing, 2015.
[28] Raghuveer, K., and R. Dayanand. "Towards handling incremental load for anomalies in near real time data warehouse." WSEAS Transactions on Systems and Control 15 (2020): 684-690.
[29] Mekterović, Igor, and Ljiljana Brkić. "Delta view generation for incremental loading of large dimensions in a data warehouse." In 2015 38th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1417-1422. IEEE, 2015.
[30] Simitsis, Alkis, Panos Vassiliadis, and Timos Sellis. "Optimizing ETL processes in data warehouses." In 21st International Conference on Data Engineering (ICDE'05), pp. 564-575. Ieee, 2005.
