Uber’s Ceilometer: Automating Infrastructure Benchmarking at Scale

Benchmarking Beyond the Application Layer: How Uber Evaluates Infrastructure Changes and Cloud Skus

Uber has launched Ceilometer, an internal framework automating infrastructure performance benchmarking beyond application metrics, enabling consistent evaluation of cloud SKUs and infrastructure updates. The system standardizes testing across servers, workloads, and environments, supporting Uber’s large-scale, heterogeneous infrastructure.

Historically, infrastructure benchmarking at Uber was a manual and fragmented process, leading to inconsistent results and hindering efficient validation of infrastructure changes. Ceilometer addresses this by providing a centralized platform for automated benchmark orchestration, execution, and analysis.

Why This Matters

Traditional application-level monitoring often obscures underlying infrastructure bottlenecks which can cause subtle performance regressions costing significant operational expenses. At Uber’s scale, even minor performance differences can translate to substantial costs when multiplied across thousands of servers and services. Ceilometer addresses these limitations by providing direct infrastructure performance signals.

Key Insights

• Fragmented Benchmarking: Uber previously relied on ad-hoc scripts and spreadsheets.
• Distributed System: Ceilometer coordinates benchmark execution across dedicated machines.
• Workload Diversity: Supports synthetic benchmarks (SPEC, NetPerf, FIO) and integration with Odin/Ballast for stateful/stateless services.

Practical Applications

• Use Case: Uber utilizes Ceilometer to qualify new cloud SKUs before onboarding, saving resources and optimizing costs.
• Pitfall: Relying solely on application-level metrics can mask underlying infrastructure issues, leading to performance degradation and increased operational costs.

References:

• https://www.infoq.com/news/2025/12/uber-infrastructure-benchmarking/
• https://www.linkedin.com/posts/navkankani_ubers-adaptive-benchmarking-framework-models-activity-7145191412913367040/