From Noise to Signal: How to Design Log Queries That Actually Catch Incidents Before Users Do

Why Bother With Log Queries at All?

Imagine a security guard on night patrol in a large office complex. She doesn't just stand at the front door; she walks the corridors and notices what's changed since yesterday — a door that should be locked but isn't, a window left open, a light on where nobody should be. Log queries do the same thing for a running software system. They walk the corridors of your service, hour by hour, and surface what has changed.

A developer writes them to debug a failing feature after the fact. An on-call engineer writes them to catch an incident while it is still unfolding. The technique is the same; the goal and the urgency are different.

It is one of the first practical skills anyone picks up in a modern engineering role, long before they touch anything glamorous like distributed tracing or chaos engineering. Almost every incident review, every postmortem, and every performance investigation begins the same way — by asking the logs a well-shaped question.

The Core Problem: Noise Drowns Signal

Most systems produce enormous volumes of logs. A modest web app can easily emit millions of lines a day. The vast majority are routine — successful requests, health checks, background jobs. This is noise.

Buried inside is signal — the small handful of events that indicate something is actually wrong, or about to be.

A beginner's instinct is to search for the word "error." This is a fine start and a terrible finish, for three reasons:

• Not all errors matter. Many are expected — a user typed a wrong password, a client cancelled a request.
• Not all incidents produce errors. A service can silently slow down, or a queue can silently stop processing, without ever logging the word.
• Volume changes everything. One error at 3 AM might be nothing. A hundred in a minute is an outage.

Good log queries aren't really about finding error messages. They're about detecting unusual patterns fast enough to act on them.

Four Principles That Turn Noise Into Signal

1. Start from user impact, not error text

Before you write a query, ask: "What would a user notice if this went wrong?"

• Checkout is broken → users can't complete a purchase.
• Login is broken → authentication requests are failing.
• The app is slow → response times are climbing.

Now translate that into a log query. Instead of searching for "NullPointerException" (which may or may not affect users), search for something like:

textservice:checkout status:500 endpoint:/api/orders

You're measuring the thing users feel, not the thing developers happen to log.

2. Count rates, not single events

A single failed request is almost never an incident. A sudden spike is.

New learners often build queries that fire on any occurrence of a keyword. This creates alert fatigue — dozens of notifications a day, none of which matter, until the one that does gets ignored.

Better: query for a rate or ratio.

• "More than 50 500-errors per minute from the checkout service."
• "Error rate above 2% of total requests, sustained for 5 minutes."
• "Zero successful logins in the last 3 minutes, when we normally see hundreds."

The last one is powerful. The absence of expected activity is often a stronger signal than the presence of errors. If your payment service usually processes 200 transactions a minute and suddenly processes zero, something is wrong even if no error was ever logged.

3. Establish a baseline before you alert

You cannot know what's abnormal without knowing what's normal.

Spend an afternoon just looking. Pick your most important service. Chart its logs over a week. What does a normal Tuesday morning look like? A Sunday night? When does traffic peak? What's the usual error rate — 0.1%? 0.5%?

Once you know the shape of normal, deviations become obvious. Your queries can then be phrased as "significantly different from baseline" rather than fixed thresholds that need constant tuning.

4. Correlate across services

Modern applications are made of many small pieces talking to each other. An incident rarely lives in one service's logs.

A useful pattern: when service A starts failing, check if service B changed behavior just before.

Example: your API is throwing timeouts. Instead of just looking at API logs, query the database logs for the same time window. Slow queries? Connection pool exhausted? Now you've moved from symptom to cause.

A Worked Example

Say you run a small e-commerce site. You want to catch checkout failures before Twitter does.

A beginner query:

text"error" AND "checkout"

This will match harmless things (a validation error where a user forgot a field) and miss real ones (a payment gateway silently returning empty responses).

A better query, built with the four principles:

textservice:checkout
AND (status:>=500 OR event:payment_timeout OR event:gateway_no_response)
AND @timestamp:[now-5m TO now]
| stats count() by endpoint
| where count > baseline_avg * 3

In plain English: "In the checkout service, count events that indicate real user-facing failure over the last 5 minutes, grouped by endpoint. Alert me when any endpoint is failing three times more than its normal rate."

You've moved from "did the word error appear" to "is a specific user-facing capability degrading faster than usual." That's the noise-to-signal transition.

Common Beginner Traps

• Alerting on every error. You'll stop reading the alerts within a week. Alert on rates, ratios, and sustained anomalies.
• Ignoring log levels. DEBUG and INFO are for humans reading after the fact. WARN and ERROR are for investigating. FATAL means something crashed hard.
• Not adding context to logs. If your logs don't include a request ID, a user ID, or a service name, correlation is nearly impossible. Push for structured logs (JSON) early.
• Searching without a time window. "Show me all errors" on a system with a year of logs is meaningless. Always bound queries by time; "last 15 minutes" is a good default.
• Copy-pasting queries without understanding them. Grab a colleague's query by all means — then read it line by line. That's how you learn.

The Mindset Shift

The goal isn't to look at logs. The goal is to have logs tell you things you didn't already know, before anyone else notices.

Every good log query is really a hypothesis: "If X is going wrong, this pattern will appear." When the pattern appears, you investigate. When it doesn't, you refine the hypothesis.

Do this consistently, and you'll stop finding out about outages from users. That's the whole game.

Key Points

Further Reading: Google SRE Book (sre.google/books), Charity Majors "Observability: A Manifesto", Elastic Observability Docs, Grafana Loki Best Practices.