Databricks redesigned its network configuration delivery system using an event-driven architecture that decouples the critical serving path from upstream dependencies. Instead of synchronously calling multiple upstream services during cluster startup, the new system subscribes to change events via a message queue, pre-computes configurations asynchronously in the background, and stores the results in a snapshot store, so serverless clusters retrieve their network configuration with a single storage read. A periodic reconciler re-syncs all workspaces to maintain eventual consistency. The shift cut 99th-percentile RPC latency from roughly 5 seconds to 125 milliseconds, raised service availability to 99.99%, and reduced upstream call volume by 86%.
