Research Article
BibTex RIS Cite

Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli

Year 2025, Volume: 40 Issue: 3, 1647 - 1658

Mehmet Ulvi Şimşek , Nuh Ali Akgül , Murat Akın

https://doi.org/10.17341/gazimmfd.1333162

Abstract

Günümüzde sensör sayısının artması ile birlikte yüksek hızda, çeşitlilikte ve hacimde veri üretilmektedir. Üretilen yüksek hızda ve farklı kaynaklardan gelen verinin birlikte analiz edilmesi önem arz etmektedir. Bu noktada büyük veri sistemleri katmanlı mimarisi ile birlikte çözümler sunmaktadır. Her bir katmanda farklı uygulamalar çalışmakta ve birbirleri ile iletişim kurarak çalışmaktadırlar. Bu çalışma, kubernetes mimarisi kullanılarak sensör verilerinin birleştirilmesi, yapay zeka yöntemleri ile anomali tespiti ve anlık verilerin işlenmesine yönelik model sunmaktadır. Önerilen sistem modeli verilerin işlenmesi ve birleştirilmesi, yapay zekâ tabanlı model geliştirilmesi ve görselleştirme aşamalarından oluşmaktadır. Bu noktada kubernetes mimarisi ile birlikte orkestrasyon işlemi sağlanarak açık kaynak kodlu uygulamalar aracılığı ile veri birleştirme, işleme ve görselleştirme işlemleri dağıtık, hata toleranslı ve etkin kaynak yönetimi sağlayacak şekilde oluşturulmuştur. Anomali tespit işlemi için ise yapay zekâ algoritmalarından LSTM, GRU ve Conv1d algoritmaları kullanılarak karşılaştırmalı analiz yapılmıştır. Sistemin normal durumuna, eğitim ve anlık veri analizi aşamasındaki durumlarına ilişkin işlemci kullanım oranları karşılaştırılarak sonuçlar sunulmuştur. Sonuç olarak önerilen kubernetes tabanlı anomali tespit modeli ile baştan sona veri toplama aşamasından başlayarak veri işleme ve görselleştirme aşamalarına ilişkin bir sistem modeli gerçekleştirilmiştir. Bu bağlamda önerilen sistem modelinin akış verilerini birleştirerek analizinde dağıtık işlem gerçekleştirme ve hata toleranslı şekilde işlemleri gerçekleştirdiği görülmüştür.

Keywords

Trafik Verisi Anomali Tespiti Büyük Veri Kubernetes Derin Öğrenme

References

  • 1. Habeeb R.A.A., Nasaruddin F., Gani A., Hashem I.A.T., Ahmed E., Imran M., Real-time big data processing for anomaly detection: A survey, Int. J. Inf. Manage., 45, 289-307, 2019.
  • 2. Bakshi K., Considerations for big data: Architecture and approach, 2012 IEEE Aerospace Conference, Big Sky - USA, 1-7, 2012.
  • 3. Morales F.A., Ramírez J.M., Ramos E.A., A mathematical assessment of the isolation random forest method for anomaly detection in big data, Math. Methods Appl. Sci., 46 (1), 1156-1177, 2023.
  • 4. Michalik P., Štofa J., Zolotova I., Concept definition for big data architecture in the education system, 2014 IEEE 12th International Symposium on Applied Machine Intelligence and Informatics (SAMI), Herl'any- Slovakia, 331-334, 2014.
  • 5. Wang J., Yang Y., Wang T., Sherratt R.S., Zhang J., Big data service architecture: A survey, J. Internet Technol., 21 (2), 393-405, 2020.
  • 6. Benhlima L. et al., Big data management for healthcare systems: Architecture, requirements and implementation, Adv. Bioinf., 2018, 2018.
  • 7. Hajjaji Y., Boulila W., Farah I.R., Romdhani I., Hussain A., Big data and IoT-based applications in smart environments: A systematic review, Comput. Sci. Rev., 39, 100318, 2021.
  • 8. Kune R., Konugurthi P.K., Agarwal A., Chillarige R.R., Buyya R., The anatomy of big data computing, Softw. Pract. Exp., 46 (1), 79-105, 2016.
  • 9. Praveena M.A., Bharathi B., A survey paper on big data analytics, 2017 International Conference on Information Communication and Embedded Systems (ICICES), Chennai - India, 1–9, 2017.
  • 10. Ma X., Wu J., Xue S., Yang J., Zhou C., Sheng Q.Z., Xiong H., Akoglu L., A comprehensive survey on graph anomaly detection with deep learning, IEEE Trans. Knowl. Data Eng., 2021.
  • 11. Rettig L., Khayati M., Cudré-Mauroux P., Piórkowski M., Online anomaly detection over big data streams, Appl. Data Sci.: Lessons Learned Data-Driven Bus., 289-312, 2019.
  • 12. Stojanovic L., Dinic M., Stojanovic N., Stojadinovic A., Big-data driven anomaly detection in industry (4.0): An approach and a case study, 2016 IEEE International Conference on Big Data (Big Data), Washington DC - USA, 1647-1652, 2016.
  • 13. Al Jallad K., Aljnidi M., Desouki M.S., Anomaly detection optimization using big data and deep learning to reduce false-positive, J. Big Data, 7 (1), 1-12, 2020.
  • 14. Zhou X., Hu Y., Liang W., Ma J., Jin Q., Variational LSTM enhanced anomaly detection for industrial big data, IEEE Trans. Ind. Inform., 17 (5), 3469-3477, 2020.
  • 15. Parwez M.S., Rawat D.B., Garuba M., Big data analytics for user-activity analysis and user-anomaly detection in mobile wireless network, IEEE Trans. Ind. Inf., 13 (4), 2058–2065, 2017.
  • 16. Karatepe I.A., Zeydan E., Anomaly detection in cellular network data using big data analytics European Wireless 2014; 20th European Wireless Conference, Barcelona - Spain, 1–5, 2014.
  • 17. Senjab K., Abbas S., Ahmed N., Khan A.U.R., A survey of Kubernetes scheduling algorithms, J. Cloud Comput., 12 (1), 87, 2023.
  • 18. Shamim S.I., Gibson J.A., Morrison P., Rahman A., Benefits, challenges, and research topics: A multi-vocal literature review of Kubernetes, arXiv preprint arXiv:2211.07032, 2022.
  • 19. Xu Z., Gong Y., Zhou Y., Bao Q., Qian W., Enhancing Kubernetes automated scheduling with deep learning and reinforcement techniques for large-scale cloud computing optimization, arXiv preprint arXiv:2403.07905, 2024.
  • 20. Dang-Quang N.M., Yoo M., Deep learning-based autoscaling using bidirectional long short-term memory for Kubernetes, Appl. Sci., 11 (9), 3835, 2021.
  • 21. Jung G., Ha H., Lee S., Anomaly detection of facilities and non-disruptive operation of smart factory using Kubernetes, J. Inf. Process. Syst., 17 (6), 2021.
  • 22. Hernández-Rivas A., Morales-Rocha V., Ruiz-Hernández O., Big data platform as a service for anomaly detection, Data Analytics and Computational Intelligence: Novel Models, Algorithms and Applications, Springer Nature, Switzerland, 2023.
  • 23. Muralidharan S., Song G., Ko H., Monitoring and managing IoT applications in smart cities using Kubernetes, Cloud Comput., 11, 2019.
  • 24. Chen Q., He Y., Yu G., Xu C., Liu M., Li Z., KBMP: Kubernetes-orchestrated IoT online battery monitoring platform, IEEE Internet Things J.,11 (14) 2024.
  • 25. I.C.P. Dataset. http://iot.ee.surrey.ac.uk:8080/. Erişim tarihi Mayıs 17, 2023.
  • 26. Ali M.I., Gao F., Mileo A., Citybench: A configurable benchmark to evaluate RSP engines using smart city datasets, The Semantic Web-ISWC 2015: 14th Int. Semantic Web Conf., Springer, 374-389, 2015.
  • 27. Smagulova K., James A.P., A survey on LSTM memristive neural network architectures and applications, Eur. Phys. J. Spec. Top., 228 (10), 2313-2324, 2019.
  • 28. Akin M., Sagiroglu S., Short term traffic speed prediction with RNN method for roads characterized by density based clustering method, Journal of the Faculty of Engineering and Architecture of Gazi University, 37 (2), 581-593, 2022.
  • 29. Wang Y., A new concept using LSTM neural networks for dynamic system identification, 2017 American Control Conference (ACC), Seattle - USA, 5324–5329, 2017.
  • 30. Cho K., Van Merriënboer B., Gulcehre C., Bahdanau D., Bougareş F., Schwenk H., Bengio Y., Learning phrase representations using RNN encoder-decoder for statistical machine translation, Empirical Methods Nat. Lang. Process. (EMNLP), 2014.
  • 31. Chung J., Gulcehre C., Cho K., Bengio Y., Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555, 2014.
  • 32. Lawal A., Rehman S., Alhems L.M., Alam M.M., Wind speed prediction using hybrid 1D CNN and BLSTM network, IEEE Access, 9, 156672-156679, 2021.
  • 33. Xiao L., Zhang L., Niu F., Su X., Song W., Remaining useful life prediction of wind turbine generator based on 1D-CNN and Bi-LSTM, Int. J. Fatigue, 163, 107051, 2022.
  • 34. Kiranyaz S., Avci O., Abdeljaber O., Ince T., Gabbouj M., Inman D.J., 1D convolutional neural networks and applications: A survey, Mech. Syst. Signal Process., 151, 107398, 2021.
  • 35. Wnęk K., Boryło P., A data processing and distribution system based on Apache Nifi, Photonics, 10 (2), 210, 2023.
  • 36. Manaouil K., Lebre A., Kubernetes and the edge?, Doktora Tezi, Inria Rennes-Bretagne Atlantique, 2020.
  • 37. Jeffery A., Howard H., Mortier R., Rearchitecting Kubernetes for the edge, Proceedings of the 4th International Workshop on Edge Systems, Analytics and Networking (EdgeSys '21), New York -USA, 7-12, 2021.
  • 38. Vayghan L.A., Saied M.A., Toeroe M., Khendek F., A Kubernetes controller for managing the availability of elastic microservice-based stateful applications, J. Syst. Softw., 175, 110924, 2021.
  • 39. Kubernetes Documentation-Cloud Controller Manager. http://web.archive.org/web/20230604021425/https://kubernetes.io/docs/concepts/architecture/cloud-controller/. Erişim tarihi Haziran 12, 2023.
  • 40. Johansson B., Rågberger M., Nolte T., Papadopoulos A.V., Kubernetes orchestration of high availability distributed control system, 2022 IEEE International Conference on Industrial Technology (ICIT), Shanghai - China, 1-8, 2022.
  • 41. Braei M., Wagner S., Anomaly detection in univariate time-series: A survey on the state-of-the-art, arXiv preprint arXiv:2004.00433, 2020.

Year 2025, Volume: 40 Issue: 3, 1647 - 1658

Mehmet Ulvi Şimşek , Nuh Ali Akgül , Murat Akın

https://doi.org/10.17341/gazimmfd.1333162

Abstract

References

  • 1. Habeeb R.A.A., Nasaruddin F., Gani A., Hashem I.A.T., Ahmed E., Imran M., Real-time big data processing for anomaly detection: A survey, Int. J. Inf. Manage., 45, 289-307, 2019.
  • 2. Bakshi K., Considerations for big data: Architecture and approach, 2012 IEEE Aerospace Conference, Big Sky - USA, 1-7, 2012.
  • 3. Morales F.A., Ramírez J.M., Ramos E.A., A mathematical assessment of the isolation random forest method for anomaly detection in big data, Math. Methods Appl. Sci., 46 (1), 1156-1177, 2023.
  • 4. Michalik P., Štofa J., Zolotova I., Concept definition for big data architecture in the education system, 2014 IEEE 12th International Symposium on Applied Machine Intelligence and Informatics (SAMI), Herl'any- Slovakia, 331-334, 2014.
  • 5. Wang J., Yang Y., Wang T., Sherratt R.S., Zhang J., Big data service architecture: A survey, J. Internet Technol., 21 (2), 393-405, 2020.
  • 6. Benhlima L. et al., Big data management for healthcare systems: Architecture, requirements and implementation, Adv. Bioinf., 2018, 2018.
  • 7. Hajjaji Y., Boulila W., Farah I.R., Romdhani I., Hussain A., Big data and IoT-based applications in smart environments: A systematic review, Comput. Sci. Rev., 39, 100318, 2021.
  • 8. Kune R., Konugurthi P.K., Agarwal A., Chillarige R.R., Buyya R., The anatomy of big data computing, Softw. Pract. Exp., 46 (1), 79-105, 2016.
  • 9. Praveena M.A., Bharathi B., A survey paper on big data analytics, 2017 International Conference on Information Communication and Embedded Systems (ICICES), Chennai - India, 1–9, 2017.
  • 10. Ma X., Wu J., Xue S., Yang J., Zhou C., Sheng Q.Z., Xiong H., Akoglu L., A comprehensive survey on graph anomaly detection with deep learning, IEEE Trans. Knowl. Data Eng., 2021.
  • 11. Rettig L., Khayati M., Cudré-Mauroux P., Piórkowski M., Online anomaly detection over big data streams, Appl. Data Sci.: Lessons Learned Data-Driven Bus., 289-312, 2019.
  • 12. Stojanovic L., Dinic M., Stojanovic N., Stojadinovic A., Big-data driven anomaly detection in industry (4.0): An approach and a case study, 2016 IEEE International Conference on Big Data (Big Data), Washington DC - USA, 1647-1652, 2016.
  • 13. Al Jallad K., Aljnidi M., Desouki M.S., Anomaly detection optimization using big data and deep learning to reduce false-positive, J. Big Data, 7 (1), 1-12, 2020.
  • 14. Zhou X., Hu Y., Liang W., Ma J., Jin Q., Variational LSTM enhanced anomaly detection for industrial big data, IEEE Trans. Ind. Inform., 17 (5), 3469-3477, 2020.
  • 15. Parwez M.S., Rawat D.B., Garuba M., Big data analytics for user-activity analysis and user-anomaly detection in mobile wireless network, IEEE Trans. Ind. Inf., 13 (4), 2058–2065, 2017.
  • 16. Karatepe I.A., Zeydan E., Anomaly detection in cellular network data using big data analytics European Wireless 2014; 20th European Wireless Conference, Barcelona - Spain, 1–5, 2014.
  • 17. Senjab K., Abbas S., Ahmed N., Khan A.U.R., A survey of Kubernetes scheduling algorithms, J. Cloud Comput., 12 (1), 87, 2023.
  • 18. Shamim S.I., Gibson J.A., Morrison P., Rahman A., Benefits, challenges, and research topics: A multi-vocal literature review of Kubernetes, arXiv preprint arXiv:2211.07032, 2022.
  • 19. Xu Z., Gong Y., Zhou Y., Bao Q., Qian W., Enhancing Kubernetes automated scheduling with deep learning and reinforcement techniques for large-scale cloud computing optimization, arXiv preprint arXiv:2403.07905, 2024.
  • 20. Dang-Quang N.M., Yoo M., Deep learning-based autoscaling using bidirectional long short-term memory for Kubernetes, Appl. Sci., 11 (9), 3835, 2021.
  • 21. Jung G., Ha H., Lee S., Anomaly detection of facilities and non-disruptive operation of smart factory using Kubernetes, J. Inf. Process. Syst., 17 (6), 2021.
  • 22. Hernández-Rivas A., Morales-Rocha V., Ruiz-Hernández O., Big data platform as a service for anomaly detection, Data Analytics and Computational Intelligence: Novel Models, Algorithms and Applications, Springer Nature, Switzerland, 2023.
  • 23. Muralidharan S., Song G., Ko H., Monitoring and managing IoT applications in smart cities using Kubernetes, Cloud Comput., 11, 2019.
  • 24. Chen Q., He Y., Yu G., Xu C., Liu M., Li Z., KBMP: Kubernetes-orchestrated IoT online battery monitoring platform, IEEE Internet Things J.,11 (14) 2024.
  • 25. I.C.P. Dataset. http://iot.ee.surrey.ac.uk:8080/. Erişim tarihi Mayıs 17, 2023.
  • 26. Ali M.I., Gao F., Mileo A., Citybench: A configurable benchmark to evaluate RSP engines using smart city datasets, The Semantic Web-ISWC 2015: 14th Int. Semantic Web Conf., Springer, 374-389, 2015.
  • 27. Smagulova K., James A.P., A survey on LSTM memristive neural network architectures and applications, Eur. Phys. J. Spec. Top., 228 (10), 2313-2324, 2019.
  • 28. Akin M., Sagiroglu S., Short term traffic speed prediction with RNN method for roads characterized by density based clustering method, Journal of the Faculty of Engineering and Architecture of Gazi University, 37 (2), 581-593, 2022.
  • 29. Wang Y., A new concept using LSTM neural networks for dynamic system identification, 2017 American Control Conference (ACC), Seattle - USA, 5324–5329, 2017.
  • 30. Cho K., Van Merriënboer B., Gulcehre C., Bahdanau D., Bougareş F., Schwenk H., Bengio Y., Learning phrase representations using RNN encoder-decoder for statistical machine translation, Empirical Methods Nat. Lang. Process. (EMNLP), 2014.
  • 31. Chung J., Gulcehre C., Cho K., Bengio Y., Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555, 2014.
  • 32. Lawal A., Rehman S., Alhems L.M., Alam M.M., Wind speed prediction using hybrid 1D CNN and BLSTM network, IEEE Access, 9, 156672-156679, 2021.
  • 33. Xiao L., Zhang L., Niu F., Su X., Song W., Remaining useful life prediction of wind turbine generator based on 1D-CNN and Bi-LSTM, Int. J. Fatigue, 163, 107051, 2022.
  • 34. Kiranyaz S., Avci O., Abdeljaber O., Ince T., Gabbouj M., Inman D.J., 1D convolutional neural networks and applications: A survey, Mech. Syst. Signal Process., 151, 107398, 2021.
  • 35. Wnęk K., Boryło P., A data processing and distribution system based on Apache Nifi, Photonics, 10 (2), 210, 2023.
  • 36. Manaouil K., Lebre A., Kubernetes and the edge?, Doktora Tezi, Inria Rennes-Bretagne Atlantique, 2020.
  • 37. Jeffery A., Howard H., Mortier R., Rearchitecting Kubernetes for the edge, Proceedings of the 4th International Workshop on Edge Systems, Analytics and Networking (EdgeSys '21), New York -USA, 7-12, 2021.
  • 38. Vayghan L.A., Saied M.A., Toeroe M., Khendek F., A Kubernetes controller for managing the availability of elastic microservice-based stateful applications, J. Syst. Softw., 175, 110924, 2021.
  • 39. Kubernetes Documentation-Cloud Controller Manager. http://web.archive.org/web/20230604021425/https://kubernetes.io/docs/concepts/architecture/cloud-controller/. Erişim tarihi Haziran 12, 2023.
  • 40. Johansson B., Rågberger M., Nolte T., Papadopoulos A.V., Kubernetes orchestration of high availability distributed control system, 2022 IEEE International Conference on Industrial Technology (ICIT), Shanghai - China, 1-8, 2022.
  • 41. Braei M., Wagner S., Anomaly detection in univariate time-series: A survey on the state-of-the-art, arXiv preprint arXiv:2004.00433, 2020.
There are 41 citations in total.

Details

Primary Language Turkish
Subjects Deep Learning
Journal Section Makaleler
Authors

Mehmet Ulvi Şimşek 0000-0001-8017-4753

Nuh Ali Akgül 0009-0003-0918-3448

Murat Akın 0000-0003-0001-1036

Early Pub Date May 13, 2025
Publication Date
Submission Date July 26, 2023
Acceptance Date January 8, 2025
Published in Issue Year 2025 Volume: 40 Issue: 3

Cite

APA Şimşek, M. U., Akgül, N. A., & Akın, M. (2025). Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, 40(3), 1647-1658. https://doi.org/10.17341/gazimmfd.1333162
AMA Şimşek MU, Akgül NA, Akın M. Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli. GUMMFD. May 2025;40(3):1647-1658. doi:10.17341/gazimmfd.1333162
Chicago Şimşek, Mehmet Ulvi, Nuh Ali Akgül, and Murat Akın. “Kubernetes Mimarisinde büyük Veri Perspektifinden Derin öğrenme yöntemleriyle Anomali Tespit Modeli”. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi 40, no. 3 (May 2025): 1647-58. https://doi.org/10.17341/gazimmfd.1333162.
EndNote Şimşek MU, Akgül NA, Akın M (May 1, 2025) Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi 40 3 1647–1658.
IEEE M. U. Şimşek, N. A. Akgül, and M. Akın, “Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli”, GUMMFD, vol. 40, no. 3, pp. 1647–1658, 2025, doi: 10.17341/gazimmfd.1333162.
ISNAD Şimşek, Mehmet Ulvi et al. “Kubernetes Mimarisinde büyük Veri Perspektifinden Derin öğrenme yöntemleriyle Anomali Tespit Modeli”. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi 40/3 (May 2025), 1647-1658. https://doi.org/10.17341/gazimmfd.1333162.
JAMA Şimşek MU, Akgül NA, Akın M. Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli. GUMMFD. 2025;40:1647–1658.
MLA Şimşek, Mehmet Ulvi et al. “Kubernetes Mimarisinde büyük Veri Perspektifinden Derin öğrenme yöntemleriyle Anomali Tespit Modeli”. Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi, vol. 40, no. 3, 2025, pp. 1647-58, doi:10.17341/gazimmfd.1333162.
Vancouver Şimşek MU, Akgül NA, Akın M. Kubernetes mimarisinde büyük veri perspektifinden derin öğrenme yöntemleriyle anomali tespit modeli. GUMMFD. 2025;40(3):1647-58.