Every benchmark gives you a number.
PINNACLE gives you an answer.
Signal65 PINNACLE is the agentic AI benchmark that measures correct work. It scores models, GPUs, CPUs, and full systems on whether the job comes out right, how fast, and what it costs, using real multi step enterprise work rather than synthetic tasks.
Outcomes, not output
Scored by code, not opinion
Deterministic, code verified scoring against an answer key the generator creates at runtime. No model judges, no human raters, no scoring drift anywhere in the path.
Nothing to memorize, nothing to tune against
Every run procedurally regenerates its own sandbox and its own answer key. There is no fixed corpus to train on and no static target to optimize toward.
Live agents, not replayed traces
Every run is a live agentic session with the answer key generated before the agent starts and held outside the sandbox, which is what makes grading possible at all. Agents explore, choose their own path, and take as many steps as they need.
Graded on real work
Scenarios are built from the work activities the US Department of Labor uses to describe occupations, and every run produces a full set of deliverables that has to be right.
FEATURED
Correct work, measured.
What the first Signal65 PINNACLE results say about agentic AI in the enterprise.
Read about itThe Results
Which model? Which silicon? Which solution?
Leaderboards rank models.
PINNACLE clears them for work.
Signal65 PINNACLE puts every model through the same multi step enterprise jobs, working from the same handbooks, producing the same deliverables, graded the same way. Every model is tested under both real world messy data and tightly controlled scope, and the score reflects both, so no model earns its ranking purely in a clean room.
Signal65 PINNACLE · Model Score
Model Score against the tokens it takes
Every PINNACLE-scored model, placed by what it gets right against what it costs to say it. Up is smarter — the PINNACLE Model Score, with Gemma-4-31B anchored at 1,000. Right is terser — correct answers per million generated tokens, in our testing. Up and to the right is the frontier. Hover any point for the full readout.
PINNACLE Model Score vs token efficiency
Reading the points: color = size bucket (the chips above are the key) · slate = open-weights with no published size. Thinking and instruct builds score separately, so a model can appear twice.
The Metrics
How the
score is built.
PINNACLE Model Score
The headline. One number per model, combining both test conditions across all personas, normalized to a fixed reference so scores stay comparable across releases.
Persona scores
Five enterprise roles, each weighting the components by how that role actually works. A service agent is graded mostly on refusing to invent answers. A strategy role is graded mostly on multi source synthesis.
Components
Each persona combines agentic task completion under both conditions, plus three retrieval measures covering grounding, fabrication resistance, and aggregation.
PINNACLE Intelligence Throughput
The productivity composite, sitting alongside rather than inside the Model Score. Quality and speed weighted three to one, with per task success compounded across workflow depth.
Tokens are a unit.
Correct work is a measure.
PINNACLE Intelligence Throughput combines quality, speed, and per task success, and the weighting is the point. Quality carries three times the weight of speed. Being correct is most often the ultimate enterprise concern and there is always a cost to failure, while speed is not zeroed out because the ability to economically host and scale a solution also matters. Three to one is our editorial judgment about the common enterprise reality.
A three times speed advantage becomes a 1.7 times productivity advantage. Scalability is rewarded. Quality cannot be bought cheaply with it.
Speed and throughput.
Measured on a loaded box, joined through tokens rather than seconds.
No Measured Duration
No published number rests on a measured duration. An elapsed time carries the box, the load, and the thermal state it was recorded under, and none of that ports to another configuration. What ports is the token cost of the work and the rate at which a platform produces tokens.
Decode Bound
The work is decode bound, so a decode rate is the portable number. On these agentic tasks a model reads roughly two orders of magnitude faster than it writes in our testing, and caching makes the reading nearly free, so a single blended rate would be wrong by construction.
Three Gates
The reported figure is the peak aggregate throughput the box reaches while every agent decodes at 10 tokens per second or better and time to first token stays inside a stated ceiling. The floor sets a usefulness bar so no platform wins by degrading its agents. The ceiling catches admission queuing, where a serving stack accepts agents it is not yet running and per agent decode looks compliant for the ones that are.
The stack is capacity.
Capacity you have already paid for, decided by software.
The serving framework, and how well it matches the accelerator's own kernels, decides how much of a box a model actually reaches. The same model on the same node holds a materially different number of agents depending on which engine serves it, and that difference costs nothing in capital. Signal65 PINNACLE measures the serving stack as its own axis: same model, same node, same workload, each engine at its own best known configuration, framework and version published. Correctness is scored on every run, because capacity bought with worse answers is not capacity. This is a supplementary view at launch, and a published axis as coverage grows.
Signal65 PINNACLE · Serving Engines
SGLang against vLLM: the engine is part of the result
Same model, same box, same ~90K-token agentic workload — two serving engines, each tuned to its own best configuration, in our testing. Correctness rides beside every number, because a throughput figure from an engine that is failing tasks is not comparable to one that is not. Pick the GPU family and model; hover anything for the full readout.
Each engine at its own best known configuration on this box · R_m = peak aggregate decode among rungs meeting the SLA
Peak aggregate decode — tok/s
Concurrent agents sustained at the 10 tok/s SLA
Aggregate decode throughput as concurrency rises · the large point is each engine's R_m
Reading the points: filled = rung passes the per-agent floor · open = rung fails it (the box is thrashing) · large = the engine's peak sustained aggregate. Hover or tap any point for the full rung readout.
All sweep rungs for this selection
A Reference Model
Anchors the quality scale. Held constant across releases so a score means something specific rather than something relative to whatever else was tested that quarter.
A Reference Box
Anchors the speed scale. A single platform on which the rate term is measured for every model, so model to model speed comparisons are not confounded by hardware.
Both are named in the whitepaper and in the methodology detail, and neither appears in the charts on this page, which lead with frontier models.
Every model is tested under both conditions, because agents fail differently depending on which one they meet in production. The PINNACLE Model Score reflects both.
Messy Data
Real enterprise conditions. Ambiguous instructions, distracting and duplicated files, and rules that have to be applied correctly to data that is actively confusing.
Controlled Scope
A bounded deployment. Clean inputs, an explicit procedure, and a long multi step job that has to be executed end to end without dropping a step.
Both conditions are hard. Controlled scope removes the ambiguity, not the work.
The Method
Our
Methodology
PICARD
Execution Harness
It builds a complete working environment for every run, generates the answer key alongside it and holds it outside the agent sandbox, and verifies the result in code. Because the environment is generated fresh each time, the space of possible task configurations is effectively unbounded and there is nothing static for a model to have memorized.
KAMI
Agentic Task Completion
Scenarios drawn from occupational work activities, run as long multi step jobs with policy handbooks to interpret, governing rules to apply, and a full set of work products to deliver.
RIKER
Grounding & Fabrication
Measures whether a model answers from what it was given, and whether it declines to invent an answer when it does not have one. Published separately with results across a large model set.
The Live Simulation Callout
The agents do not know they are being tested.
Every run is a live agentic session inside a complete working environment. The agent explores, decides its own approach, calls real tools against real generated data, takes as many steps as it needs, and produces its own trace. Two models running the same scenario generate different traces, different token counts, and different round counts, exactly as they would in production.
You cannot grade a recording. A trace contains what happened, not what should have happened. No answer key was ever generated for a recorded session, so there is nothing to check an output against. That is why benchmarks built on recordings report throughput and never quality. Measuring correct work requires generating the work, because the answer has to be created in the same moment as the environment.
The alternative approach in this space records real sessions and replays them. Replay pins every model to the same sequence of steps and the same number of rounds. That undermines the intelligence measurement, because no model is choosing its own path, and it corrupts the scalability measurement, because a model that would naturally take 50% more rounds and far more tokens is prevented from showing it. The differences that matter get suppressed by the method itself.
Occupational Grounding
The scenarios are not invented. They are built from the Generalized Work Activities in the O*NET database maintained by the US Department of Labor, which is the taxonomy the department uses to describe what work consists of across the occupational landscape. Occupational analysts defined the work. Signal65 PINNACLE tests agents against it.
Of the 41 work activities in that taxonomy, Signal65 identified 26 that current models can plausibly perform and that can be simulated and scored deterministically. Those 26 are the Signal65 Agent-Testable Work Activities. The remaining 15 are outside the reach of any software benchmark, requiring physical embodiment or turning on interpersonal outcomes where the human response is the point. Scenarios are designed to exercise the testable set, and those activities appear across a wide range of occupational roles.
Signal65 Agent-Testable Work Activities
26
of the 41 activities in the taxonomy, selected through two stated gates
The department defined the activities. Signal65 determined which are testable. That determination is Signal65 methodology, not a Department of Labor classification.
Read the methodology whitepaperGovernance
The rules are published.
So is every configuration.
Signal65 publishes the governance rules that determine how results are produced, reviewed, corrected, and refreshed, alongside the full configuration behind every published number. Vendors may review results for factual accuracy before publication and hold no approval right over what is published.
Read the governance documentThe Roadmap
What
Comes Next
Models and Scenarios
More scenarios across more of the testable work activity set, a broader model set, and day zero coverage of new model releases across every platform in the lab.
Silicon and Precision
More accelerators from both vendors as systems reach the lab, and node shapes beyond the single node the launch results measure. The same model at more than one precision, since reduced precision is where a platform wins on cost and a throughput number cannot say whether the answers survived it. Serving frameworks measured the same way, so the effect of the stack on throughput and on answer quality becomes a published axis rather than a supplementary view.
Decomposed Workflows
Real deployments rarely give one model the whole job. A capable orchestrator routes work to smaller models running narrow, well specified subtasks, and a model that fails a full scenario outright may be entirely adequate in that role. Measuring the decomposed shape is the next expansion of the scenario suite.
Distributed and CPU Side Measurement
Agentic deployments split inference and tool execution across separate infrastructure, and the tool side of that split runs on CPUs no accelerator benchmark accounts for. Measuring that split, and the CPU side of agentic work specifically, is the next expansion of the harness.
Cluster Acceptance
Grading a delivered cluster rather than a component, the services around the accelerators included, and tracking how that grade moves from first light to steady state, which is where a meaningful share of deployed capacity is actually lost.
New Results
News
and Contact
Latest Results and Reports
Report
Signal65 PINNACLE, Measuring Correct Work. A methodology for benchmarking agentic AI in the enterprise.
Governance
Signal65 PINNACLE Benchmark Governance. How results are produced, reviewed, corrected, and refreshed.
Article
Correct work, measured. What the first Signal65 PINNACLE results say about agentic AI in the enterprise.
Get in touch
Whether you have a question about the methodology, want to see results for a platform we have not covered, or are weighing an evaluation of your own systems, we are happy to talk. Signal65 runs PINNACLE on customer hardware or in the cloud across models, accelerators, complete platforms, and deployed clusters.