A Comparative Empirical Study of Messaging Primitives for Enterprise-Scale Event-Driven Microservices: EventBridge, SQS, SNS, and Apache Kafka under a Unified Decision Framework
DOI:
https://doi.org/10.63282/3050-922X.IJERET-V4I3P116Keywords:
Event-Driven Architecture, Messaging Primitives, Microservices, Apache Kafka, Amazon Eventbridge, Empirical Software EngineeringAbstract
Event-driven architecture has become a dominant integration style for enterprise microservice platforms because synchronous request-response meshes accumulate latency, failure modes, and coupling that constrain independent service evolution. The selection of an appropriate messaging primitive, however, remains under-studied: vendor documentation tends to advocate single-primitive solutions, while academic treatments often abstract over the operational and economic differences that dominate real deployment choices. This paper presents a comparative empirical study of four messaging primitives commonly composed in enterprise-scale event-driven backbones: Amazon EventBridge for content-based routing, Amazon Simple Queue Service for reliable point-to-point work distribution, Amazon Simple Notification Service for fan-out pub-sub, and Apache Kafka for high-throughput streaming with replay. The contribution is a workload-to-primitive decision framework, calibrated against measurements collected from three representative production deployments spanning regulated enterprise workloads, and an analysis of the dimensions on which the primitives differ in practice. The methodology characterizes workloads along interaction shape, ordering requirement, replay requirement, throughput, and acceptable tail latency, and defines a measurement protocol for end-to-end latency, sustainable throughput, ordering guarantees, and per-million-message cost. The data show that no single primitive dominates: EventBridge minimizes integration cost for low-to-moderate volumes with heterogeneous consumers, Kafka is the only studied primitive that sustains five-figure message rates with replay, and SQS-FIFO remains the most cost-effective choice when strict ordering is required at moderate throughput. The findings have implications for architects designing event-driven backbones in any large enterprise context where multiple interaction shapes must coexist on a shared substrate.
References
[1] Hohpe, G. and Woolf, B. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley, 2003. | https://scholar.google.com/scholar?q=Hohpe, G. and Woolf, B. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley, 2003.
[2] Newman, S. Building Microservices, Second Edition. O'Reilly Media, 2021.| https://scholar.google.com/scholar?q=Newman, S. Building Microservices, Second Edition. O'Reilly Media, 2021.
[3] Richardson, C. Microservices Patterns: With Examples in Java. Manning Publications, 2018.| https://scholar.google.com/scholar?q=Richardson, C. Microservices Patterns: With Examples in Java. Manning Publications, 2018.
[4] Stopford, B. Designing Event-Driven Systems. O'Reilly Media, 2018. https://scholar.google.com/scholar?q=Stopford, B. Designing Event-Driven Systems. O'Reilly Media, 2018.
[5] Kreps, J., Narkhede, N., and Rao, J. Kafka: a distributed messaging system for log processing. Proceedings of NetDB, 2011. | https://scholar.google.com/scholar?q=Kreps, J., Narkhede, N., and Rao, J. Kafka: a distributed messaging system for log processing. Proceedings of NetDB, 2011.
[6] Wang, G., Koshy, J., Subramanian, S., Paramasivam, K., Zadka, M., Narkhede, N., Rao, J., Kreps, J., and Stein, J. Building a replicated logging system with Apache Kafka. Proceedings of the VLDB Endowment, 8(12), 2015, pp. 1654-1655. | https://scholar.google.com/scholar?q=Wang, G., Koshy, J., Subramanian, S., Paramasivam, K., Zadka, M., Narkhede, N., Rao, J., Kreps, J., and Stein, J. Building a replicated logging system with Apache Kafka. Proceedings of the VLDB Endowm
[7] Helland, P. Life beyond Distributed Transactions: an Apostate's Opinion. Proceedings of CIDR, 2007. | https://scholar.google.com/scholar?q=Helland, P. Life beyond Distributed Transactions: an Apostate's Opinion. Proceedings of CIDR, 2007.
[8] Vogels, W. Eventually Consistent. Communications of the ACM, 52(1), 2009, pp. 40-44. | https://scholar.google.com/scholar?q=Vogels, W. Eventually Consistent. Communications of the ACM, 52(1), 2009, pp. 40-44.
[9] Lamport, L. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7), 1978, pp. 558-565. | https://scholar.google.com/scholar?q=Lamport, L. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7), 1978, pp. 558-565.
[10] Birman, K. P. and Joseph, T. A. Reliable communication in the presence of failures. ACM Transactions on Computer Systems, 5(1), 1987, pp. 47-76. | https://scholar.google.com/scholar?q=Birman, K. P. and Joseph, T. A. Reliable communication in the presence of failures. ACM Transactions on Computer Systems, 5(1), 1987, pp. 47-76.
[11] Chandy, K. M. and Lamport, L. Distributed snapshots: determining global states of distributed systems. ACM Transactions on Computer Systems, 3(1), 1985, pp. 63-75. | https://scholar.google.com/scholar?q=Chandy, K. M. and Lamport, L. Distributed snapshots: determining global states of distributed systems. ACM Transactions on Computer Systems, 3(1), 1985, pp. 63-75.
[12] Dean, J. and Barroso, L. A. The tail at scale. Communications of the ACM, 56(2), 2013, pp. 74-80. | https://scholar.google.com/scholar?q=Dean, J. and Barroso, L. A. The tail at scale. Communications of the ACM, 56(2), 2013, pp. 74-80.
[13] Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernandez-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., and Whittle, S. The Dataflow Model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment, 8(12), 2015, pp. 1792-1803. | https://scholar.google.com/scholar?q=Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernandez-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., and Whittle, S. The Dataflow Model: a practical approach to b
[14] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink: stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4), 2015, pp. 28-38. | https://scholar.google.com/scholar?q=Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink: stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4), 2015, pp. 28-38.
[15] Kleppmann, M. Designing Data-Intensive Applications. O'Reilly Media, 2017. | https://scholar.google.com/scholar?q=Kleppmann, M. Designing Data-Intensive Applications. O'Reilly Media, 2017.
[16] Sigelman, B. H., Barroso, L. A., Burrows, M., Stephenson, P., Plakal, M., Beaver, D., Jaspan, S., and Shanbhag, C. Dapper, a large-scale distributed systems tracing infrastructure. Google Technical Report, 2010. | https://scholar.google.com/scholar?q=Sigelman, B. H., Barroso, L. A., Burrows, M., Stephenson, P., Plakal, M., Beaver, D., Jaspan, S., and Shanbhag, C. Dapper, a large-scale distributed systems tracing infrastructure. Google Technical Re
[17] Fowler, M. and Lewis, J. Microservices: a definition of this new architectural term. martinfowler.com, 2014. | https://scholar.google.com/scholar?q=Fowler, M. and Lewis, J. Microservices: a definition of this new architectural term. martinfowler.com, 2014.| https://martinfowler.com/articles/microservices.html
[18] Amazon Web Services. Amazon EventBridge Developer Guide. | https://scholar.google.com/scholar?q=Amazon Web Services. Amazon EventBridge Developer Guide. | https://docs.aws.amazon.com/eventbridge/
[19] Amazon Web Services. Amazon Simple Queue Service Developer Guide.| https://scholar.google.com/scholar?q=Amazon Web Services. Amazon Simple Queue Service Developer Guide.| https://docs.aws.amazon.com/sqs/
[20] Amazon Web Services. Amazon Simple Notification Service Developer Guide.| https://scholar.google.com/scholar?q=Amazon Web Services. Amazon Simple Notification Service Developer Guide. | https://docs.aws.amazon.com/sns/
[21] Amazon Web Services. Amazon Managed Streaming for Apache Kafka Developer Guide.| https://scholar.google.com/scholar?q=Amazon Web Services. Amazon Managed Streaming for Apache Kafka Developer Guide. | https://docs.aws.amazon.com/msk/
[22] Apache Software Foundation. Apache Kafka documentation. | https://scholar.google.com/scholar?q=Apache Software Foundation. Apache Kafka documentation. | https://kafka.apache.org/documentation/
[23] Confluent Inc. Schema Registry documentation. | https://scholar.google.com/scholar?q=Confluent Inc. Schema Registry documentation. | https://docs.confluent.io/platform/current/schema-registry/index.html
[24] OpenTelemetry Project. OpenTelemetry specification. | https://scholar.google.com/scholar?q=OpenTelemetry Project. OpenTelemetry specification. | https://opentelemetry.io/docs/specs/otel/
[25] CloudEvents Working Group, Cloud Native Computing Foundation. CloudEvents v1.0 specification. | https://scholar.google.com/scholar?q=CloudEvents Working Group, Cloud Native Computing Foundation. CloudEvents v1.0 specification. https://github.com/cloudevents/spec
[26] Amazon Web Services. AWS Well-Architected Framework: Reliability Pillar. AWS Whitepaper, 2023. https://scholar.google.com/scholar?q=Amazon Web Services. AWS Well-Architected Framework: Reliability Pillar. AWS Whitepaper, 2023.
