CCDCOE Technology Branch researchers Markus Kont and Mauno Pihelgas present a novel real-time pattern matching engine that functions as an IDS for logs. The paper demonstrates that a small custom-built streaming tool can easily handle a task usually relegated to much larger databases. It also presents a technical specification for implementing a streaming Sigma rule engine in Golang, something that to the best of the authors’ knowledge did not exist during Crossed Swords 2020.
Pattern-based intrusion detection systems (IDS) form the cornerstone of network security monitoring (NSM) and threat detection. Signatures are the staple of threat detection. The Sigma rule format has emerged in recent years to fulfil this role for event logs and has become increasingly popular in security operations and threat hunting communities. The Public Sigma project provides rules and tooling for conversion into various SIEM provider alerting formats. It is not a match engine, so users are still expected to deploy a log management solution. In many cases, this is not ideal and real-time integration into existing log streams would be preferred.
NATO CCDCOE has organised Exercise Crossed Swords annually since 2015. It aims to train the red team, whereas the yellow team is tasked with providing real-time feedback to players. The authors have implemented an experimental rule engine in Golang and made the source code publicly available. Since then, that engine has been rewritten so it would serve as a better reference for anyone who needs to implement such a solution in their own environment. This paper provides a detailed technical outline of the implementation. Performance benchmarks were conducted to assess potential limitations of the approach and propose further developments.
Markus Kont is a Researcher in the Technology branch of the NATO Cooperative Cyber Defence Centre of Excellence (CCDCOE). His area of expertise is packet capture and log processing, DevOps tools and techniques and data science. His current work involves researching stream processing techniques and he is responsible for teaching network security monitoring tools in the CCDCOE. He holds a Master’s degree in Cyber Security from Tallinn University of Technology.
Mauno Pihelgas has been a Researcher in the Technology branch of the CCDCOE since 2013. His area of expertise is monitoring, data mining and situational awareness. In addition to being a GIAC GMON Continuous Monitoring Certified Professional, he is also a Red Hat Certified System Administrator, Red Hat Certified Engineer and a Red Hat Certified Specialist in Ansible Automation. Mauno holds a Master’s of Science degree in Cyber Security and is pursuing a PhD at the Tallinn University of Technology.
This research paper is an independent product of the CCDCOE and does not represent the official policy or position of NATO or any of the CCDCOE´s Sponsoring Nations. The NATO Cooperative Cyber Defence Centre of Excellence (NATO CCDCOE) is a NATO-accredited knowledge hub, research institution, and training and exercise facility. The Tallinn-based international military organisation focuses on interdisciplinary applied research, as well as consultations, training and exercises in the field of cyber security.