SIEM and Log Analysis Basics: Detecting Threats with Elastic Stack

A SIEM (Security Information and Event Management) system centralizes logs from all your infrastructure and helps detect threats automatically. Enterprise SIEMs cost a fortune, but you can build basic capabilities with Elastic Stack for free. Here's how.

Prerequisites

Docker Compose (for local/VPS setup)
Linux servers to monitor
8GB+ RAM for the Elastic Stack server

Elastic Stack Setup

# compose.yml
services:
 
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "127.0.0.1:9200:9200"
    healthcheck:
      test: ["CMD-SHELL", "curl -su elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health | grep -q '\"status\":\"green\"\\|\"status\":\"yellow\"'"]
      interval: 30s
      retries: 5
 
  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    restart: unless-stopped
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
    ports:
      - "127.0.0.1:5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy
 
  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    restart: unless-stopped
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5044:5044"   # Beats input
      - "514:514/udp" # Syslog input
 
volumes:
  es_data:

Filebeat on Monitored Servers

Install Filebeat on every server you want to monitor:

# Install on Ubuntu/Debian
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.13.0-amd64.deb
sudo dpkg -i filebeat-8.13.0-amd64.deb

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/auth.log
      - /var/log/syslog
    tags: ["linux", "auth"]
 
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    tags: ["nginx"]
    json.keys_under_root: true
 
# Use modules for structured log parsing
filebeat.modules:
  - module: system
    auth:
      enabled: true
    syslog:
      enabled: true
  - module: nginx
    access:
      enabled: true
    error:
      enabled: true
 
output.logstash:
  hosts: ["siem-server:5044"]

sudo systemctl enable filebeat
sudo systemctl start filebeat

Logstash Pipeline

# logstash/pipeline/main.conf
input {
  beats {
    port => 5044
  }
  udp {
    port => 514
    codec => plain
    tags => ["syslog"]
  }
}
 
filter {
  # Parse SSH auth failures
  if "auth" in [tags] {
    grok {
      match => { "message" => "Failed password for %{USER:ssh_user} from %{IP:src_ip} port %{INT:src_port}" }
      tag_on_failure => []
    }
 
    if [src_ip] {
      geoip { source => "src_ip" }
    }
  }
 
  # Parse nginx access logs
  if "nginx" in [tags] and [log][file][path] =~ "access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    mutate {
      convert => { "response" => "integer" }
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    index => "siem-%{[tags][0]}-%{+YYYY.MM.dd}"
  }
}

Detection Rules in Kibana

Navigate to Security → Rules → Create new rule

Rule 1: Brute force SSH detection

{
  "name": "SSH Brute Force Attempt",
  "type": "threshold",
  "query": "message: \"Failed password\" AND tags: auth",
  "threshold": {
    "field": "src_ip",
    "value": 10
  },
  "time_window": "5m",
  "severity": "high",
  "actions": ["slack-alert"]
}

Rule 2: Successful login after multiple failures

{
  "name": "Successful Login After Brute Force",
  "type": "eql",
  "query": "sequence with maxspan=5m\n  [authentication where event.outcome == \"failure\"] with runs=5\n  [authentication where event.outcome == \"success\"]",
  "severity": "critical"
}

Rule 3: Privilege escalation

Query: message: "sudo" AND (message: "COMMAND" OR message: "authentication failure")
Alert when: event count > 3 in 10m per user

Key Security Dashboards to Build

In Kibana Dashboards, create visualizations for:

Auth events over time — bar chart of login successes vs. failures
Top source IPs for auth failures — data table, sort by count
Geographic map of SSH attempts — using geoip enrichment
HTTP error codes over time — 4xx/5xx rate from nginx logs
New processes spawned — if using Elastic Agent with process monitoring

What to Monitor (Minimum Viable SIEM)

| Log Source | Key Events to Alert On | |-----------|------------------------| | SSH (/var/log/auth.log) | >10 failures in 5min, successful login from new country | | Sudo (/var/log/auth.log) | Any sudo command by non-admin users | | Nginx access | 4xx spike (>100/min), 5xx errors | | Cron (/var/log/syslog) | New crontab entries (crontab -e by non-root) | | Package manager | apt install / yum install during off-hours | | Firewall (ufw.log) | Port scans (>20 blocked IPs per minute) |

Common Pitfalls

Elasticsearch without authentication: default is open in older versions — always enable xpack.security.enabled=true
No log rotation: Elasticsearch indices grow indefinitely — set up Index Lifecycle Management (ILM) policies
Too many low-severity alerts: alert fatigue = ignored alerts. Start with high/critical only
Single node without replication: your SIEM data can't protect itself from disk failure — use snapshots to S3/GCS

Resources

Prerequisites

Docker Compose (for local/VPS setup)
Linux servers to monitor
8GB+ RAM for the Elastic Stack server

Elastic Stack Setup

# compose.yml
services:
 
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    restart: unless-stopped
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    ports:
      - "127.0.0.1:9200:9200"
    healthcheck:
      test: ["CMD-SHELL", "curl -su elastic:${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health | grep -q '\"status\":\"green\"\\|\"status\":\"yellow\"'"]
      interval: 30s
      retries: 5
 
  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    restart: unless-stopped
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
    ports:
      - "127.0.0.1:5601:5601"
    depends_on:
      elasticsearch:
        condition: service_healthy
 
  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    restart: unless-stopped
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5044:5044"   # Beats input
      - "514:514/udp" # Syslog input
 
volumes:
  es_data:

Filebeat on Monitored Servers

Install Filebeat on every server you want to monitor:

# Install on Ubuntu/Debian
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.13.0-amd64.deb
sudo dpkg -i filebeat-8.13.0-amd64.deb

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/auth.log
      - /var/log/syslog
    tags: ["linux", "auth"]
 
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log
    tags: ["nginx"]
    json.keys_under_root: true
 
# Use modules for structured log parsing
filebeat.modules:
  - module: system
    auth:
      enabled: true
    syslog:
      enabled: true
  - module: nginx
    access:
      enabled: true
    error:
      enabled: true
 
output.logstash:
  hosts: ["siem-server:5044"]

sudo systemctl enable filebeat
sudo systemctl start filebeat

Logstash Pipeline

# logstash/pipeline/main.conf
input {
  beats {
    port => 5044
  }
  udp {
    port => 514
    codec => plain
    tags => ["syslog"]
  }
}
 
filter {
  # Parse SSH auth failures
  if "auth" in [tags] {
    grok {
      match => { "message" => "Failed password for %{USER:ssh_user} from %{IP:src_ip} port %{INT:src_port}" }
      tag_on_failure => []
    }
 
    if [src_ip] {
      geoip { source => "src_ip" }
    }
  }
 
  # Parse nginx access logs
  if "nginx" in [tags] and [log][file][path] =~ "access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    mutate {
      convert => { "response" => "integer" }
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    index => "siem-%{[tags][0]}-%{+YYYY.MM.dd}"
  }
}

Detection Rules in Kibana

Navigate to Security → Rules → Create new rule

Rule 1: Brute force SSH detection

{
  "name": "SSH Brute Force Attempt",
  "type": "threshold",
  "query": "message: \"Failed password\" AND tags: auth",
  "threshold": {
    "field": "src_ip",
    "value": 10
  },
  "time_window": "5m",
  "severity": "high",
  "actions": ["slack-alert"]
}

Rule 2: Successful login after multiple failures

{
  "name": "Successful Login After Brute Force",
  "type": "eql",
  "query": "sequence with maxspan=5m\n  [authentication where event.outcome == \"failure\"] with runs=5\n  [authentication where event.outcome == \"success\"]",
  "severity": "critical"
}

Rule 3: Privilege escalation

Query: message: "sudo" AND (message: "COMMAND" OR message: "authentication failure")
Alert when: event count > 3 in 10m per user

Key Security Dashboards to Build

In Kibana Dashboards, create visualizations for:

Auth events over time — bar chart of login successes vs. failures
Top source IPs for auth failures — data table, sort by count
Geographic map of SSH attempts — using geoip enrichment
HTTP error codes over time — 4xx/5xx rate from nginx logs
New processes spawned — if using Elastic Agent with process monitoring

What to Monitor (Minimum Viable SIEM)

Common Pitfalls

Elasticsearch without authentication: default is open in older versions — always enable xpack.security.enabled=true
No log rotation: Elasticsearch indices grow indefinitely — set up Index Lifecycle Management (ILM) policies
Too many low-severity alerts: alert fatigue = ignored alerts. Start with high/critical only
Single node without replication: your SIEM data can't protect itself from disk failure — use snapshots to S3/GCS

SIEM and Log Analysis Basics: Detecting Threats with Elastic Stack

Prerequisites

Elastic Stack Setup

Filebeat on Monitored Servers

Logstash Pipeline

Detection Rules in Kibana

Key Security Dashboards to Build

What to Monitor (Minimum Viable SIEM)

Common Pitfalls

Resources

Related posts

SIEM and Log Analysis Basics: Detecting Threats with Elastic Stack

Prerequisites

Elastic Stack Setup

Filebeat on Monitored Servers

Logstash Pipeline

Detection Rules in Kibana

Key Security Dashboards to Build

What to Monitor (Minimum Viable SIEM)

Common Pitfalls

Resources

Related posts