System Design Practice: Metric Monitoring System

Functional Requirements

FR1: Platform can ingest metrics (CPU, memory, latency, custom counters) from services.
FR2: Users can query and visualize metrics on dashboards with filters, aggregations, and time ranges.
FR3: Users can define alert rules with thresholds over time windows, e.g., “alert if P99 latency > 500ms for 5min”.
FR4: Users can receive notifications when alerts fire, e.g., email, Slack.

Scale: 500K servers, 5M metrics per second, ~1GB/s raw ingestion.

Non-Functional Requirements

At first, I tried evaluating all 8 areas , but calls for the top 3-5 that apply. Makes sense as some like “environment concerns” aren’t interesting because we’re designing a server application that can be as powerful as our wallets allow.

System should be highly available. Dashboards can tolerate eventual consistency.

Initially had this the other way round where I did not want to create false alarms. However, a metric monitoring system that goes down during incidents is not useful at all.

Scalability. Need to scale both writes (500K servers each emitting 100 metric data points every 10s gets as 5M metrics/second at peak) and reads (dashboard queries should evaluate in seconds, and alerts should fire in less than a minute). A minute or so allows you to accumulate enough data in order to “see” an event, and then fire an alert.

The point here is to quantify with ballparks. “A firehose of incoming data” doesn’t help you pick between options. Do some napkin math.

The 1min for alert latency is flexible. For example, Amazon has a consistently large number of orders. This allows them to track the number of ms since their last order and fire an alert almost immediately when something’s wrong.

Security. Metrics are meant to be low cardinality non-PII data. That said, integrations with alert services like Slack involves PII that comes with additional security requirements, e.g., access control. Authoring metrics also needs access control as that’s used to determine actions downstream, e.g., when to ping on-call folks.

Fault Tolerance. To a point. Past some time \(t\), recovery doesn’t help as old data is not useful in issuing metrics updates. There’s a time window that we care about.

did not consider security and fault tolerance in their top concerns.

Network delays (or even retried payloads) mean metrics don’t always come in order. At the scale of 5M metrics/second, this is a worthwhile problem to handle.

Missed this in the quiz. In the future, think of the core operations, and what you need to consider about those core operations.

Core Entities

A label is a key-value pair attached to a metric that lets you slice and filter, e.g., host="server-1".

A metric is a named measurement with labels and a value at some time \(t\), e.g., cpu_usage{host="server-1", region="us-west"} = 0.75.

A series is a full sequence of (timestamp, value) pairs for one specific metric + label combination, e.g., cpu_usage{host="server-1"} over time, cpu_usage{host="server-2"} over time, etc.

An alert rule combines a metric query (e.g., average CPU in us-east), threshold (above 90%), and duration (for 5min), and triggers notifications when violated.

A dashboard is a collection of panels, each displaying a query result as a chart or table.

I also had these core entities:

User representation, e.g., login, and access control lists for metrics/alert ownership.
Notification sinks. Where the alerts are delivered to, e.g., email group(s), Slack channel, etc.

Data Flow

The core flows:

Services generate metric data points and send them to our platform.
The platform ingests, validates, and stores metrics as time-series data.
Users query stored metrics through dashboards (and adhoc queries).
The platform periodically evaluates alert rules against stored metrics.
On violation of an alert condition, the platform sends a notification to the configured channel.

… where steps 1-2 are write-heavy and continuous, step 3 is read-heavy and bursty (engineers debugging incidents), and steps 4-5 need to be reliable above all else.

I should consider the core data flow before the non-functional requirements as the former informs the latter.

System API

Ingesting metrics is a high-volume operation. Protobufs are ideal for the wire protocol. Using JSON here for ease in writing:

POST /v1/metrics/ingest
{
  name: string,
  value: number,
  timestamp: number,
  labels: {name: string, value: string | number}[],
}[]

Initially did not batch POST /v1/metrics/ingest. Consider that ingestion is a high volume operation. You’d typically have SDKs installed on the services that batch metrics and send them in one payload. Sending individual metrics is taxing both on the services and the platform.

In the ingestion stage, using protobufs probably comes with providing services with an SDK for emitting metrics. OpenTelemetry helps with this by being a middle layer between telemetry-emitting services and observability backends (like Jaeger, Prometheus, etc.) . IRL, I’d design the platform against the OTel specifications so that it’s one less thing for services to know about.

Querying metrics benefits from a domain-specific language. Prometheus Query Language (PromQL) is an industry example on querying time-series data, e.g., http_requests_total{job="apiserver", handler="/api/comments"}[5m] to return time series for the http_requests_total metric with the given job and handler labels over the past 5 minutes; topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m]))) for the top 3 CPU users grouped by application and process type. Our platform can use something similar, e.g., GET /metrics/query?query=avg(cpu_usage{region="us-east"})&start=A&end=B&step=60.

Prior experience with could have helped here. I initially had:

POST /v1/metrics/query
{
  name: string,
  labels: {name: string, value: string | number}[],
  aggregation: {
    type: "average" | "percentile" | ...,
    ... # params specific to aggregation type
  },
  startTime: number,
  endTime?: number, # Omitting this triggers SSE events for latest updates
}

… which is a rudimentary version of a querying DSL. A proper DSL shifts complexity away from the request payload. Complexity of the payload was a reason why I opted to use POST as a GET-like request, because GET requests shouldn’t have bodies.

The DSL specification also requires front-end support for forming the queries and backend support for evaluating them.

POST /v1/alerts/rules
{
  name: "High CPU alert",
  query: "avg(cpu_usage{region='us-east'}) > 0.9", # Using the DSL
  duration: number,
  notificationChannels: {
    type: "email" | "slack" | "pagerDuty",
    ... # params specific to notification channel type
  }[]
}

References

Design a Metrics Monitoring Platform like Datadog. Stefan Mai. www.hellointerview.com . Accessed Jul 8, 2026.
Query examples | Prometheus. prometheus.io . Accessed Jul 9, 2026.
GET request method - HTTP | MDN. developer.mozilla.org . Accessed Jul 9, 2026.
What is OpenTelemetry? | OpenTelemetry. opentelemetry.io . Accessed Jul 9, 2026.