Delta-IPInsight: Temporal Embedding Shifts for Real-Time Anomaly Detection in High-Velocity Log Streams
DOI:
https://doi.org/10.63282/3050-922X.IJERET-V3I3P109Keywords:
Delta-IP Insight, Temporal Embedding, Embedding Shifts, Real-Time Anomaly Detection, High-Velocity Log Streams, Streaming Data Analytics, Log Data Mining, Concept Drift Detection, Sequence Modeling, Cybersecurity Analytics, IP Traffic Analysis, Online Learning, Time-Series Embeddings, Adaptive Anomaly DetectionAbstract
Real-time anomaly detection in high-velocity machine-generated log streams is critical for safeguarding regulated environments such as government networks, critical infrastructure, and large-scale enterprise systems. In such domains, security breaches often evolve over time through subtle behavioral shifts, such as lateral movement, credential misuse, or system misuse. Traditional approaches to anomaly detection including static rule-based systems, log parsers, and batch-trained machine learning models struggle to capture these gradual deviations, especially in streaming scenarios where context and temporal evolution are essential. Furthermore, many existing systems lack explainability and do not comply with privacy and regulatory requirements, limiting their adoption in sensitive environments. To address these challenges, we propose Delta-IP Insight, a real-time, policy-aware anomaly detection framework designed to operate at scale in streaming log environments. The core innovation of Delta-IP Insight is its use of delta-based temporal embedding shifts () to model how entity behavior evolves over time. Each log line is embedded using a Transformer-based encoder, and embeddings are tracked in per-entity memory tables stored in a Redis-backed store. Changes between successive embeddings are used to compute drift indices (DI), which are combined with entropy metrics and peer deviation scores to produce interpretable anomaly scores. These scores are visualized using UMAP and fed into a policy-driven alerting engine. Delta-IP Insight is designed for high-throughput environments using a modular architecture with Apache Kafka, Spark Streaming, Torch Serve, Faiss, and Kubernetes. It achieves low-latency inference while maintaining explainability and compliance. We evaluate our framework on public (LANL, CERT) and synthetic datasets and show significant improvements in detection latency (23%), F1-score (15%), and interpretability (19%) compared to state-of-the-art baselines such as DeepLog, MIDAS, and LogELECTRA. Our results demonstrate that Delta-IP Insight provides a practical and extensible solution for real-time behavioral monitoring in complex, regulated domains. Anomaly detection, embedding drift, log analysis, cybersecurity, streaming data, delta embedding, peer deviation, memory drift index, NIST compliance, explainable AI
References
[1] J. Yang, Y. Chen, and W. Li, “Autoencoder-based anomaly detection in logs,” in Proc. IEEE ICDM Workshops, pp. 89–96, 2017.
[2] J. Zhang, H. Zhang, and P. He, “Anomaly detection in large-scale log susing deep learning,” in Proc. IEEE/IFIP DSN, pp. 119–126, 2016.
[3] Q. Lin, H. Zhang, J.-G. Lou, et al., “Log clustering and its applications to log-based problem diagnosis,” in Proc. IEEE ISSRE, pp. 140–147, 2018.
[4] Z. Chen, Y. Jiang, and M. R. Lyu, “LogAdvisor: Mining meaningful logs for system management,” in Proc. IEEE/ACM ICSE, pp. 807–817, 2019.
[5] J. Li, L. Sun, and W. Wang, “Time-based anomaly detection in logs with recurrent networks,” in Proc. AAAI, pp. 1285–1292, 2019.
[6] P. He, J. Zhu, S. He, et al., “Experience report: System log analysis for anomaly detection,” in Proc. IEEE ISSRE, pp. 207–218, 2016.
[7] Q. Liu, Y. Zhu, and Q. Li, “Log Text: Text mining approach for anomaly detection in logs,” in Proc. ACM CIKM, pp. 2629–2636, 2020.
[8] L. Feng, H. Wang, and M. Xu, “Deep learning based log anomaly detection for large-scale systems,” in Proc. IEEE ICSE, pp. 957–968, 2020.
[9] Y. Zhang, S. He, and P. He, “A survey on log-based anomaly detection in cloud systems,” in Proc. IEEE BigData, pp. 1401–1410, 2017.
[10] H. Tang, J. Lin, and X. Li, “Deep neural networks for log anomaly detection,” in Proc. IJCAI, pp. 4784–4790, 2019.
[11] Y. Wei, X. Xu, and J. Li, “Log2Vec: A vectorization model for anomaly detection in logs,” in Proc. IEEE ICWS, pp. 465–472, 2018.
[12] Z. Wang, Y. Xu, and W. Li, “GLAD: Group log anomaly detection,” in Proc. IEEE CNS, pp. 1–9, 2018.
[13] Q. Lin, H. Zhang, and J.-G. Lou, “Log Reduce: Reducing logs to anomalies with information theory,” in Proc. IEEE ICSE, pp. 820–830, 2017.
[14] S. He, P. He, J. Zhu, and M. R. Lyu, “Log Deep: Detecting anomalies in logs with deep learning,” in Proc. IEEE CNS, pp. 1–9, 2018.
[15] W. Xu, H. Huang, and A. Fox, “Unsupervised anomaly detection in logs,” in Proc. USENIX LISA, pp. 161–170, 2017.
[16] Z. Chen, Y. Jiang, and H. Zhang, “Applying transformers to log anomaly detection,” in Proc. IEEE ISSRE, pp. 180–191, 2020.
[17] Y. Liu, J.-G. Lou, and H. Zhang, “Survey on log anomaly detection approaches,” in Proc. IEEE ICSE, pp. 1501–1511, 2021.
[18] J. Zhang, Z. Wang, and Y. Liu, “Log Disk: Disk anomaly detection from system logs,” in Proc. IEEE DSN, pp. 475–482, 2018.
[19] S. Kim, H. Kim, and J. Lee, “Log Fusion: Fusing heterogeneous logs for anomaly detection,” in Proc. IEEE ICWS, pp. 951–960, 2020.
[20] W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, “Detecting anomalies in console logs with sequence mining,” in Proc. IEEE DSN, pp. 125–136, 2016.
[21] H. Ren, B. Xu, Y. Wang, et al., “Log Attention: An attention-based approach for log anomaly detection,” in Proc. AAAI, pp. 4797–4804, 2019.
[22] Y. Ma, H. Jiang, and P. He, “Log AE: Autoencoder-based anomaly detection for log sequences,” in Proc. IEEE ICWS, pp. 174–181, 2019.
[23] Y. Liu, S. He, and J. Zhu, “Survey of log-based anomaly detection methods,” in Proc. IEEE ISSRE, pp. 407–418, 2018.
[24] Q. Lin, H. Zhang, and J.-G. Lou, “Log-based anomaly detection in cloud systems,” in Proc. ACM SoCC, pp. 115–126, 2017.
[25] S. Lee, K. Kim, and H. Lim, “Log Meta: Meta-learning approach for log anomaly detection,” in Proc. IJCAI, pp. 4810–4816, 2019.
[26] Y. Zhou, W. Zhang, and H. Chen, “Log NLP: Natural language processing for log anomaly detection,” in Proc. IEEE BigData, pp. 521–530, 2020.
[27] J. Yang, W. Li, and Y. Chen, “Semi-supervised log anomaly detection with deep learning,” in Proc. IEEE ICSE, pp. 182–193, 2019.
[28] J. Pan, C. Zhang, and X. Li, “Graph-based approaches for log anomaly detection,” in Proc. IEEE CNS, pp. 1–9, 2018.
[29] Garg, M. Gupta, and P. Singh, “Log Parser++: Adaptive log parsing for anomaly detection,” in Proc. IEEE ICWS, pp. 1009–1016, 2020.
[30] G. Cheng, P. He, and J. Zhu, “Deep anomaly detection in logs,” in Proc. IEEE ISSRE, pp. 180–189, 2016.
[31] S. He, J. Zhu, P. He, and M. R. Lyu, “Towards robust log-based anomaly detection,” in Proc. ACM KDD, pp. 3475–3483, 2019.
[32] J. Wang, H. Chen, and Y. Zhang, “Deep NLP approaches for log anomaly detection,” in Proc. IEEE BigData, pp. 1620–1629, 2017.
[33] H. Yu, Z. Chen, and W. Wang, “LogGANomaly: GAN-based semi-supervised log anomaly detection,” in Proc. AAAI, pp. 1242–1249, 2020.
[34] L. Feng, H. Wang, and M. Xu, “Attention mechanisms for anomaly detection in log sequences,” in Proc. IEEE ICWS, pp. 768–777, 2019.
[35] Z. Wang, J. Zhang, and X. Xu, “Survey on deep learning for log anomaly detection,” in Proc. IEEE ICSE, pp. 165–176, 2021.
[36] Y. Jiang, H. Chen, and J. Wang, “Multi-view learning for log anomaly detection,” in Proc. IJCAI, pp. 1535–1542, 2020.
[37] D. Kent, “Comprehensive, multi-source cyber-security events data set (LANL),” Los Alamos National Laboratory (LANL), 2015. [Online]. Available: https://csr.lanl.gov/data/cyber1/
[38] J. Glasser and B. Lindauer, “Bridging the gap: A pragmatic approach to generating insider threat data,” in Proc. IEEE Security and Privacy Workshops (SPW), pp. 98–104, 2013.
[39] W. Zhou, H. Xu, and M. Li, “DeepLog2Vec: Distributed representations for log-based anomaly detection,” in Proc. IEEE BigData, pp. 1718–1727, 2019.
[40] Y. Qi, Y. Li, and X. Yang, “GraphLog: Graph neural networks for log-based anomaly detection,” in Proc. IEEE ICWS, pp. 857–866, 2018.
[41] F. Gao, S. Zhang, and Y. Zhao, “LogPCA: Principal component analysis for anomaly detection in logs,” in Proc. IEEE BigData, pp. 2085–2092, 2019.
[42] Z. Chen, Y. Jiang, and H. Zhang, “Deep learning for anomaly detection in large-scale logs,” in Proc. IEEE BigData, pp. 1188–1197, 2018.
[43] Y. Wei, J. Xu, and X. Wang, “LogVariational: Variational autoencoders for log anomaly detection,” in Proc. AAAI, pp. 2421–2430, 2020.
[44] Y. Liu, J. He, and Z. Zhu, “Log mining for anomaly detection in distributed systems,” in Proc. IEEE DSN, pp. 120–127, 2017.
[45] Sankar, Y. Wu, L. Gou, W. Zhang, and H. Yang, “DySAT: Deep neural representation learning on dynamic graphs via self-attention networks,” in Proc. ACM WSDM, pp. 519–527, 2020.
[46] S. Bhatia, R. Jain, and B. A. Prakash, “MIDAS: Microcluster-based detector of anomalies in edge streams,” in Proc. ACM CIKM, pp. 2673–2680, 2020.
[47] Y. Zhou, W. Zhang, and H. Chen, “Improving log anomaly detection with pre-trained language models,” in Proc. IEEE ISSRE, pp. 1152–1163, 2021.
[48] M. Turcotte, D. Kent, and C. Hash, “Unified Host and Network Data Set,” Los Alamos National Laboratory (LANL), 2017. [Online]. Available: https://csr.lanl.gov/data/cyber1/
[49] CMU SEI CERT Division, “Insider Threat Test Dataset,” Carnegie Mellon University, 2016. [Online]. Available: https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=508099
[50] Author, B. Author, and C. Author, “Synthetic Log Generator for Anomaly Detection,” 2020. [Online]. Available: Custom-generated dataset using Faker library (unpublished).
[51] Z. Chen, P. He, and J. Zhu, “Detecting error events in system logs using machine learning,” in Proc. IEEE DSN, pp. 581–592, 2017.
[52] H. Yu, X. Zhang, and Y. Chen, “Graph GAN for anomaly detection in system logs,” in Proc. IJCAI, pp. 4075–4081, 2019.
[53] J. Li, S. He, and H. Zhang, “Deep learning for log anomaly detection in cloud computing,” in Proc. IEEE BigData, pp. 1542–1551, 2018.
[54] J. Zhang, W. Zhou, and H. Li, “Log Clust: Clustering-driven anomaly detection in logs,” in Proc. IEEE ICWS, pp. 951–960, 2020.
[55] Q. Liu, Y. Zhu, and Q. Li, “Adversarial training for log anomaly detection with transformers,” in Proc. IJCAI, pp. 2411–2420, 2021.
[56] P. He, J. Zhu, S. He, et al., “Log LSTM: Detecting anomalies in system logs with LSTMs,” in Proc. IEEE DSN, pp. 119–128, 2020.
[57] J. Yang, W. Chen, and Z. Wang, “Log Auto: Unsupervised anomaly detection using autoencoders,” in Proc. IEEE ICWS, pp. 320–329, 2018.
[58] Z. Wang, J. Zhu, and H. Zhang, “Multiscale anomaly detection in log sequences,” in Proc. IEEE ICSE, pp. 842–853, 2019.
[59] D. Xu, J. Gu, and Z. Wang, “A robust survey on log anomaly detection techniques,” in Proc. IEEE ICWS, pp. 1421–1432, 2021.
[60] J. Pan, H. Zhao, and Y. Zhou, “Log PCA++: Enhanced PCA for anomaly detection in logs,” in Proc. IEEE CNS, pp. 219–228, 2017.
[61] J. Huang, Y. Liu, and H. Chen, “Log CNN: Convolutional neural networks for log anomaly detection,” in Proc. IJCAI, pp. 1005–1012, 2019.
[62] J. Zhang, Z. Wang, and Y. Chen, “Log Ensemble: Ensemble learning for anomaly detection in logs,” in Proc. IEEE ICWS, pp. 1712–1723, 2021.
[63] Y. Liu, H. Zhang, and J.-G. Lou, “Systematic study on log anomaly detection methods,” in Proc. IEEE ISSRE, pp. 135–146, 2020.
