Scalable Data Management in Distributed Systems: A Sharding-Based Approach for Multi-Tenant Architectures
DOI:
https://doi.org/10.63282/3050-922X.IJERET-V5I3P101Keywords:
Sharding, Multi-Tenant Architecture, Scalability, Load Balancing, Data Consistency, Distributed Systems, Hybrid Sharding, Fault Tolerance, Data Migration, Performance EvaluationAbstract
In the era of big data and cloud computing, managing large volumes of data efficiently and scalably is a critical challenge for modern distributed systems, especially in multi-tenant architectures. This paper explores the concept of sharding as a powerful technique for achieving scalable data management. We delve into the theoretical foundations of sharding, its implementation in multi-tenant environments, and the associated challenges and solutions. We present a comprehensive sharding-based approach that addresses data distribution, load balancing, and data consistency. Additionally, we evaluate the performance of our proposed approach through a series of experiments and simulations. The results demonstrate significant improvements in scalability, performance, and resource utilization. This paper aims to provide a robust framework for designing and implementing scalable data management systems in distributed and multi-tenant environments
References
[1] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 51(1), 107-113.
[2] Corbett, J. C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J. J., ... & Yushprakh, A. (2012). Spanner: Google’s Globally-Distributed Database. ACM Transactions on Computer Systems (TOCS), 31(3), 1-22.
[3] Abadi, D. J., Madden, S. R., & Hwang, M. (2005). Integrating compression and execution in column-oriented database systems. Proceedings of the 2006 ACM SIGMOD international conference on Management of data, 671-682.
[4] Bailis, P., & Ghodsi, A. (2013). Eventual consistency today: limitations, extensions, and beyond. ACM Queue, 11(3), 20-50.
[5] Ongaro, D., & Ousterhout, J. (2014). In Search of an Understandable Consensus Algorithm. Proceedings of the USENIX Annual Technical Conference, 305-319.
[6] DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., ... & Vogels, W. (2007). Dynamo: Amazon’s Highly Available Key-value Store. ACM SIGOPS Operating Systems Review, 41(6), 205-220.
[7] Kulkarni, S., & Sheth, A. (2010). Cloud computing: Issues, challenges and future research directions. Proceedings of the 2010 International Conference on Cloud and Service Computing (CSC), 1-6.
[8] Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop Distributed File System. Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 1-10.
[9] Stonebraker, M., Abadi, D. J., Batkin, A., Chen, X., Cherniack, M., Ferreira, M., ... & Zdonik, S. (2007). C-Store: A Column-Oriented DBMS. Proceedings of the 33rd International Conference on Very Large Data Bases, 553-564.
[10] Zhang, Y., Li, J., & Chen, L. (2015). A survey on data sharding in cloud databases. Journal of Cloud Computing, 4(1), 1-23.
[11] https://www.clearpeaks.com/real-time-data-monitoring-using-scalable-distributed-systems/
[12] https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/approaches/storage-data
[13] https://www.pingcap.com/article/distributed-database-architecture-secure-scalable-data-management/
[14] https://daily.dev/blog/multi-tenant-database-design-patterns-2024
[15] https://tropars.github.io/downloads/lectures/LSDM/LSDM-1-introduction.pdf
[16] https://www.risein.com/blog/understanding-sharding-in-database-architecture
[17] https://www.informatik.tu-darmstadt.de/systems/teach/lectures/sdms/index.en.jsp
[18] https://ijsrcseit.com/index.php/home/article/view/CSEIT241061151
