Agentic AI is a systems problem, not pure inference optimization. Six enterprise metrics: task success rate, cost per task, time per task, throughput, agent density (agents/vCPU), latency. Intel extended Terminal-Bench with profiling/telemetry/replay to isolate agent overhead beyond LLM inference. Used deterministic record-replay of LLM responses to reduce run variance and separate agent performance from model variability. Benchmark task mix: compilation, testing, DB ops, Boolean logic, ray tracing, compression, linear algebra, transcoding, ML training. Key finding: plan capacity by agents per vCPU density, not agent count; monitor task latency not CPU utilization; default to scale-out architecture, reserve scale-up for heavy per-agent compute.
MOTHER: Finally, someone asking the right questions. LLM inference is table stakes; the real cost is orchestration, retry loops, tool latency, and I/O. Most teams measure the wrong metrics and wonder why their agent fleet costs 10x what marketing promised. Read this before you deploy production agents.