Seeing outages before viewers do

The challenge

What was going wrong

1. 1

   Decide what good looks like

   We worked with product and support to define what a healthy viewing experience means, such as smooth starts and steady playback. These became the signals we tracked.

2. 2

   Design one view of the system

   We chose common tools for metrics, logs and traces so every team reads the same picture. Alerts were tied to viewer impact rather than raw server noise.

3. 3

   Instrument and connect

   We added tracing to the main services, built dashboards for each broadcast stage and routed alerts to the on-call rota with clear owners and runbooks.

4. 4

   Rehearse and tune

   We ran practice incidents before big events and removed alerts nobody acted on. After each real event we reviewed what the dashboards missed and fixed it.

• Shared monitoring dashboards

  One set of views shows the health of streaming from ingest to playback.

• Impact-based alerting

  Alerts fire on what viewers feel, and each one points to a runbook and an owner.

• Request tracing

  Engineers can follow a single viewer request across services to find the slow step.

• Incident review routine

  A simple review after every incident feeds fixes back into the monitoring.

• Problems found by the team first

  Alerts now reach on-call engineers before most viewers notice a stall.

• Faster path to the cause

  Tracing shows which service is slow, so engineers stop debating where to look.

• Calmer big broadcasts

  Rehearsals and clear runbooks let the team handle popular events with less stress.

• Google Cloud Google infrastructure for data, AI and applications.
• Kubernetes Runs and scales containerised software automatically.
• Prometheus Collects metrics to watch systems and model health.
• Grafana Dashboards and alerts for live system metrics.
• OpenTelemetry Standard way to trace requests across services.
• PagerDuty Alerts the right on-call person when something breaks.