What happened
Databricks benchmarked coding agents on its multi-million line codebase, revealing that the Pareto frontier for quality-to-cost includes models from OpenAI, Anthropic, and open-source options like GLM 5.2. GLM 5.2 statistically tied with Anthropic's Opus 4.8 on quality, costing $1.28/task compared to Opus's $1.94/task. The analysis also found token price a poor indicator of actual costs, with larger models often more token-efficient. Harnesses significantly impacted efficiency, reducing costs by over 2x for the same model by optimising context management. Databricks plans to deploy GLM 5.2 as a daily driver.
Why it matters
Engineering teams must adopt a multi-model, multi-harness strategy to optimise coding agent costs and performance. Databricks' internal benchmark demonstrated that open models like GLM 5.2 achieve top-tier quality at significantly lower costs ($1.28/task versus $1.94/task for Opus 4.8), with token efficiency, not just token price, driving overall task costs. Harness choice critically impacts cost, reducing expenses by over 2x for identical models through better context management. Platform engineers and CTOs should implement internal benchmarking to identify optimal model-harness combinations for diverse coding tasks, moving beyond token-based cost assumptions. This follows Databricks' recent launch of Enterprise AI Agents, providing practical deployment guidance.
