Automating Threat Hunting with Sigma Rules and Elastic Stack

Introduction#

Threat hunting is a proactive security activity that involves actively searching for cyber threats that might have bypassed existing security measures. Traditional security information and event management (SIEM) systems rely heavily on predefined rules and signatures. However, advanced adversaries often employ techniques that evade these standard defenses. Sigma rules provide a generic signature format for SIEM systems, allowing security analysts to describe relevant log events in a structured manner. When combined with the Elastic Stack (Elasticsearch, Logstash, and Kibana), Sigma rules can be used to automate threat hunting, significantly improving an organization’s ability to detect and respond to sophisticated attacks. This article will guide you through the process of automating threat hunting with Sigma rules and the Elastic Stack.

Understanding Sigma Rules#

Sigma is an open standard for describing log events. It provides a structured and flexible way to define patterns that can be used to identify malicious activity. Unlike vendor-specific SIEM rules, Sigma rules are platform-agnostic, meaning they can be converted into queries for various SIEM systems, including the Elastic Stack.

A Sigma rule typically consists of the following components:

title: A descriptive name for the rule.
id: A unique identifier for the rule.
status: The current status of the rule (e.g., experimental, stable).
description: A detailed explanation of the threat or activity the rule is designed to detect.
references: Links to external resources that provide additional information about the threat.
author: The name of the rule’s creator.
date: The date the rule was created or last updated.
logsource: Specifies the type of log data the rule applies to (e.g., product: windows, service: sysmon).
detection: Contains the actual search criteria, including keywords, field mappings, and logical operators.
level: Indicates the severity of the detected event (e.g., informational, low, medium, high, critical).

Here’s an example of a simple Sigma rule to detect PowerShell execution with encoded commands:

title: PowerShell Encoded Command Detection
id: 123e4567-e89b-12d3-a456-426614174000
status: stable
description: Detects PowerShell execution with Base64 encoded commands.
references:
    - https://attack.mitre.org/techniques/T1059/001/
author: John Doe
date: 2024/05/22
logsource:
    product: windows
    service: powershell
detection:
    selection:
        EventID: 400
        ScriptBlockText|contains:
            - "-EncodedCommand"
            - "-enc"
            - "-e"
    condition: selection
level: high

Setting Up the Elastic Stack#

Before you can automate threat hunting with Sigma rules, you need to have the Elastic Stack properly configured. This involves setting up Elasticsearch, Logstash, and Kibana.

Install Elasticsearch: Elasticsearch is the core search and analytics engine. Download and install Elasticsearch from the official Elastic website. Configure the elasticsearch.yml file to set cluster name, node name, and network settings.
Install Logstash: Logstash is a data processing pipeline that ingests, transforms, and ships data to Elasticsearch. Download and install Logstash. Create a Logstash configuration file (e.g., logstash.conf) to define the input, filter, and output stages.
Example Logstash configuration:
```
input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "weblogs-%{+YYYY.MM.dd}"
  }
}
```
Install Kibana: Kibana is a data visualization and exploration tool that allows you to interact with the data stored in Elasticsearch. Download and install Kibana. Configure the kibana.yml file to connect to your Elasticsearch instance.
Ingest Logs: Configure your systems to send logs to Logstash. This can be done using various methods, such as Filebeat, Winlogbeat, or syslog. Ensure that the logs are properly parsed and indexed in Elasticsearch.

Converting Sigma Rules to Elasticsearch Queries#

To use Sigma rules with the Elastic Stack, you need to convert them into Elasticsearch queries. This can be done using tools like sigma-cli or pySigma.

Using sigma-cli:

Install sigma-cli:
```
pip install sigma-cli
```
Convert Sigma rule to Elasticsearch query:
```
sigma convert -t elasticsearch <sigma_rule.yml>
```
This command will output the Elasticsearch query equivalent to the Sigma rule.

Using pySigma:

pySigma is a Python library that provides a more programmatic way to convert and manage Sigma rules.

Install pySigma:
```
pip install pySigma
```

Write a Python script to convert Sigma rules:

from sigma.parser import SigmaParser
from sigma.backends.elasticsearch import ElasticsearchBackend

# Load the Sigma rule
with open("powershell_encoded_command.yml", "r") as f:
    rule_yaml = f.read()

# Initialize the Sigma parser
parser = SigmaParser(rule=rule_yaml, config="./sigma/config/generic/sigma_backend.yml")

# Initialize the Elasticsearch backend
backend = ElasticsearchBackend(parser.rule)

# Convert the Sigma rule to an Elasticsearch query
query = backend.convert()

print(query)

Automating Threat Hunting with Kibana#

Once you have converted your Sigma rules to Elasticsearch queries, you can use Kibana to automate threat hunting.

Create Saved Searches: In Kibana, create saved searches for each Sigma rule. Use the converted Elasticsearch query as the search query.
Create Alerts: Use Kibana’s alerting feature to create alerts based on the saved searches. Configure the alert to trigger when the number of hits exceeds a certain threshold.
- Go to Stack Management > Rules and Alerts.
- Create a new rule based on an index threshold.
- Select the index pattern that contains your logs.
- Define the threshold conditions (e.g., trigger when the count is above 0).
- Add actions, such as sending an email or creating a Slack notification.
Create Dashboards: Create dashboards to visualize the alerts and trends. This will provide a high-level overview of the security posture and help identify potential threats.

Example: Detecting Suspicious Process Execution#

Let’s consider a scenario where you want to detect suspicious process execution using a Sigma rule.

Sigma Rule:

title: Suspicious Process Execution
id: a1b2c3d4-e5f6-7890-1234-567890abcdef
status: stable
description: Detects suspicious process execution based on command-line arguments.
references:
    - https://attack.mitre.org/techniques/T1059/
author: Jane Smith
date: 2024/05/22
logsource:
    product: windows
    service: sysmon
detection:
    selection:
        EventID: 1
        CommandLine|contains:
            - "cmd.exe /c"
            - "powershell.exe -exec bypass"
            - "wmic process call create"
    condition: selection
level: medium

Converted Elasticsearch Query (using sigma-cli):

{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "event.code": "1"
          }
        },
        {
          "bool": {
            "should": [
              {
                "match": {
                  "process.command_line": "cmd.exe /c"
                }
              },
              {
                "match": {
                  "process.command_line": "powershell.exe -exec bypass"
                }
              },
              {
                "match": {
                  "process.command_line": "wmic process call create"
                }
              }
            ],
            "minimum_should_match": 1
          }
        }