Cloud vendors are very good at benchmarks. Every platform has a chart showing why its engine is faster, cheaper, or both. The problem is that those numbers are usually produced under conditions chosen by the vendor.I needed a different kind of answer. I was comparing several cloud processing options for a large data organization, and the decision could affect thousands of pipelines. I could not base that choice on a slide deck or a one-off test from a single team. I needed a benchmark that was neutral, repeatable, and easy to run across very different systems.That is what pushed me toward TPC-DS. Instead of asking which vendor had the best story, I ran the same analytical workload across multiple engines and measured the thing the business actually cared about: how much useful performance we received for the cost of running it.Why Vendor Benchmarks Are Hard to TrustVendor benchmark results are not automatically wrong. The problem is that they are optimized for a particular question. One provider may use a data layout that suits its engine. Another may tune memory, caching, or cluster size aggressively. A third may highlight only the workload sizes where it performs best.Inside a large engineering organization, the same problem can happen without any vendor involvement. Different teams run different pipelines, use different configurations, and then compare the results as if they were measuring the same thing. You end up with many data points but no common ruler.The fix is not another custom workload. The fix is standardization.Why I Chose TPC-DSTPC-DS is a decision-support benchmark designed for analytical systems. It uses a retail-style data model and a set of 99 SQL queries that cover realistic analytical patterns such as trend analysis, customer segmentation, reporting, promotions, and aggregations.What made it useful for my evaluation was not that it perfectly represented every production pipeline. No benchmark can do that. It was useful because every engine had to answer the same questions against the same data.I ran the workload at several data sizes because engine behavior changes with scale. A platform that looks excellent on a small dataset may become expensive at larger sizes. Another may have startup overhead that hurts small jobs but performs very well when the workload grows.I also kept the basic test conditions consistent. Data was stored externally in a columnar format, queries were run one at a time, and the same query set was repeated enough times to reduce the effect of random variation. The goal was not to make any engine look good. The goal was to make the comparison boringly fair.Speed Alone Is the Wrong WinnerA common benchmarking mistake is to rank systems only by runtime. That works if cost does not matter. In cloud infrastructure, cost always matters.I used a simple price-performance idea: combine the runtime of an execution with the retail cost of that execution. A fast engine that costs much more can lose to a slightly slower engine that is dramatically cheaper. The best choice depends on the balance, not on either number alone.This also prevents a common argument between engineering and finance. Engineering does not have to defend a slower system just because it is cheap, and finance does not have to accept a large premium because something is technically faster. Both sides can look at the same ratio.The Most Interesting Result Was Not One Universal WinnerThe benchmark did not tell me that one engine should replace every other engine. It showed that different engines were strong at different workload sizes.Some serverless options were especially attractive for smaller and medium jobs because they avoided the overhead of keeping dedicated capacity around. Other engines became more competitive as data size increased. The performance curve mattered more than the headline ranking.That changed the migration question. Instead of asking, 'Which engine should the whole company standardize on?' I could ask, 'Which kinds of workloads should run where?' That is a much more useful infrastructure decision.A Benchmark Becomes Valuable When You Map It Back to RealityA synthetic benchmark is only the first half of the exercise. The next step was comparing the benchmark profiles with the real population of production jobs.I looked at how existing pipelines were distributed by workload size and estimated which TPC-DS scale range best represented each group. That allowed me to project how the same portfolio might behave on different processing environments without migrating every pipeline just to learn the answer.This is where the cost opportunity became clear. A large share of the workload did not need the same runtime architecture it had historically been given. Moving the right classes of jobs toward more elastic processing could create seven-figure annual savings without rewriting the business logic inside those jobs.That point matters. Cloud cost optimization is often presented as a code-efficiency problem. Sometimes it is an architecture-placement problem. The query can stay the same while the economics change completely underneath it.What TPC-DS Still Does Not Tell YouI would not use these results as a universal ranking of cloud engines. TPC-DS uses synthetic data, and real production data is messier. Real tables have skew, odd distributions, schema changes, unexpected hotspots, and years of historical baggage.My initial comparison also focused on single-query execution rather than heavy concurrency. That is useful when you want to isolate raw performance, but shared warehouses and serverless systems are built to serve many workloads at once. Their economics can look different when dozens or hundreds of queries compete for resources.Caching is another factor. Some platforms can dramatically improve repeated analytical workloads by keeping hot data or intermediate results close to compute. A clean benchmark run may not fully capture the way a real organization benefits from that behavior over time.These are not reasons to avoid benchmarking. They are reasons to understand what your benchmark actually proves.How I Would Run the Evaluation TodayIf I were starting this exercise again, I would keep the same neutral core but expand it in three directions.First, I would add concurrency tests early. Single-query speed is useful, but modern data platforms increasingly consolidate workloads onto shared infrastructure. Testing many queries together shows whether an engine still looks efficient when the system is busy.Second, I would test newer table formats alongside the basic columnar baseline. Storage layout, metadata handling, and file pruning can change the amount of work an engine has to perform before the SQL engine even becomes the bottleneck.Third, I would rerun the benchmark regularly. Cloud engines change quickly. Pricing changes. Optimizers improve. Serverless startup times get better. The platform that was the best fit a year ago may not be the best fit now.The Bigger Lesson: Build Your Own EvidenceThe biggest lesson from this project was not about any particular cloud platform. It was about decision quality.When infrastructure choices involve large amounts of money, every vendor has an incentive to show you the comparison where it wins. Your job is not to decide which marketing team produced the most convincing graph. Your job is to create a test where every system has to play by the same rules.TPC-DS gave me that common ruler. It was not perfect, but it was repeatable, public, and close enough to real analytical work to expose meaningful differences between engines.My advice to other data teams is simple. Do not take anyone's benchmark on faith, including mine. Pick a neutral workload, run it at the scales you actually care about, measure both speed and cost, and then map the results back to your real workload mix.The best cloud engine is not the one that wins the internet benchmark. It is the one that wins your workload.