Data Pipeline

Databricks lakehouse, medallion architecture

How transportation data flows into the SDOT World Cup operations dashboard: each external feed lands in the Databricks warehouse, where it is ingested, cleansed, and modeled, then served to the dashboard by an Azure Static Web App.

DERQ

Traffic counts

Iteris ClearGuide

Travel times

Weather (NOAA)

Conditions & forecast

TOC Daily Notes

Daily operations summary

TOC Incidents

Collisions, other

Other Event Feeds

Waze, GTFS, WSDOT

Databricks Warehouse

Medallion architecture: Bronze → Silver → Gold
Raw ingestion through cleansed, conformed tables
Timeseries fact + location dimension tables
Match day vs baseline metrics
Geocoded, validated locations

SQL endpoint

Live query

Azure Static Web App

Serves this operations dashboard
Managed /api/data function
Databricks credentials held server-side
Live query on each page load

Live app

Live feed Planned ?Optional

Refresh Rates

Per feed

15-30 min