Denys Miasnychenko

Software Architect / DevOps

Nav

Head-of-line blocking: how one dead output stalled a Logstash pipeline

This is a write-up of a production incident from September 2026. A Logstash pipeline on three log shipping nodes stopped sending anything for two days, and nobody noticed: the process was running, the process check was green, and the persisted queue was filling up. The cause was one output pointing at an Elasticsearch cluster that had just been turned off. The notes apply to any single-pipeline Logstash (7.17 here) with several outputs and queue.type: persisted.

Problem

The setup is one pipeline with several outputs, chosen by conditionals on routing key, channel and application. Then one destination dies. Here it was an old Elasticsearch 6 cluster behind the log relay host, stopped for decommissioning.

The elasticsearch output retries a batch forever on connection errors. The worker thread holding that batch never finishes. The other workers pick up batches that also contain events for the dead output, and they get stuck too. Result:

Nagios only noticed when a restart left the process down (PROCS CRITICAL: 0 processes). The stall itself ran for two days unnoticed, because the process was running the whole time.

Detection

# per pipeline: throughput + queue fill
curl -s localhost:9600/_node/stats/pipelines/main | python3 -c '
import sys,json
p=json.load(sys.stdin)["pipelines"]["main"];q=p["queue"]
print("out",p["events"]["out"],"q_events",q["events_count"],
      "q_bytes",q["queue_size_in_bytes"],"/",q["max_queue_size_in_bytes"])
for o in p["plugins"]["outputs"]:
    e=o.get("events",{}); print(o["name"],o["id"][:8],"in",e.get("in"),"out",e.get("out"))'

# map output plugin ids to config lines
curl -s "localhost:9600/_node/pipelines/main?graph=true"

Signs:

Monitor events.out rate and PQ fill, not only “is the process running”.

Fix applied

  1. Remove the dead output instead of pointing it at nothing. Here the output nodes got their own config template, so the shared template used by the other Logstash hosts stayed untouched.
  2. Route leftover events to Kafka with bounded retries, so they are neither dropped silently nor allowed to block the pipeline. Tag them in filter{} (the @metadata is not serialized by the json codec), then branch in output{}:
filter {
    if [channel] != 'apm' and ([application] == "legacy-app" or ([@metadata][output][routing_key] != "main-elk" and [@metadata][output][routing_key] != "%{routing_key}")) {
        mutate { add_field => { "[@metadata][output][unrouted]" => "true" } }
        mutate { add_field => { "unrouted_routing_key" => "%{[@metadata][output][routing_key]}" } }
        if [@metadata][dead_letter_queue][reason] {
            mutate { add_field => { "unrouted_reason" => "dead-letter" } }
        } else {
            mutate { add_field => { "unrouted_reason" => "unknown-routing-key" } }
        }
    }
}

output {
    if [channel] != 'apm' {
        if [@metadata][output][unrouted] == "true" {
            kafka {
                bootstrap_servers => "kafka-1.example.com:9092"
                topic_id          => "logstash-unrouted"
                codec             => json
                retries           => 3
                retry_backoff_ms  => 1000
            }
        } else if [@metadata][output][routing_key] == "main-elk" {
            elasticsearch { hosts => ["es-main.example.com:9203"] ... }
        } else if [@metadata][output][routing_key] == "%{routing_key}" {
            elasticsearch { hosts => ["es-main.example.com:9203"] ... }
        }
    }
}
  1. Create the Kafka topic before rollout, with explicit retention. Do not rely on auto-create.

Gotchas

Rollout and backlog

Longer-term options