12. Scoring
Every agentic run is scored two ways, only one of the two enters the headline, and the composite is built from there.
12.1 Two scoring modes
Figure 6: The two scoring modes.
Every reader has met this as a grading policy. All or nothing per question against partial credit per question is a choice a teacher makes, and the two produce very different rankings over the same set of papers.
Partial credit earns its place because it measures potential rather than delivery. A model producing six correct deliverables out of seven has shipped nothing usable, and it has also demonstrated that it understood the job, located the governing rule, and executed most of it. That is a materially different proposition from a model that produced two of seven, and the difference describes what an integration team would be working with.
12.2 Why only workflow completion enters the headline
The headline PINNACLE Model Score is built from workflow completion in both environments. A deployed workflow that produces six of seven correct outputs has not shipped anything usable. Somebody still has to find the wrong one, and finding it requires redoing the work that was supposed to be delegated. Binary scoring is the view that matches what deployment requires.
Per-deliverable accuracy is measured and published alongside, never folded in. The distance between the two carries a great deal of the useful signal, which is the subject of the next section.
A run that produces seven deliverables and gets six right scores 0 on workflow completion and roughly 86% on per-deliverable accuracy. A run that gets three right scores 0 and roughly 43%. Under the headline, those two runs are identical, and under the companion measure they are not close. Both readings are true and they answer different questions.
12.3 The distance between the two modes
The gap between workflow completion and per-deliverable accuracy is an estimate of integration effort, and it is the second thing the de-aggregated structure makes available.
A model that completes few whole jobs while producing mostly correct individual deliverables is telling a buyer that the remaining distance is closable with engineering. Orchestration, validation steps, retry logic, and tighter prompting all attack exactly that gap, and a model in this position is a reasonable bet for a team willing to build around it. A model scoring poorly on both is telling them something else, because no scaffolding manufactures deliverables the model never got right.
Both numbers publish. Neither is folded into the other, and the distance between them is reported as a figure in its own right.
12.4 How the composite is built
Persona scores are a weighted geometric mean of the five measured capabilities using that persona weights. The overall PINNACLE Model Score is an unweighted geometric mean across the five personas.
The geometric mean is the standard aggregation for composite benchmarks and it is used here for the same reasons it is used elsewhere. It treats percentage improvements symmetrically, it prevents one high scoring component from dominating, and it preserves relative rankings regardless of the reference chosen. Leaving the outer mean unweighted means a model has to be competent across all five personas, since catastrophic weakness in one cannot be bought back by excellence in another.
12.5 The reference model
K is set so that the reference model scores exactly 1000 on every persona and 1000 overall. The reference is Gemma-4-31B-it with thinking enabled, chosen on three grounds. It is a widely used open model with an established following. It comes from a major research lab rather than an obscure one. And during data collection it landed near the midpoint of the agentic score distribution, which puts the anchor in the middle of the field rather than at either end.
The last of those matters most for a scale. An anchor at the frontier compresses every other model into a narrow band below 1000, and an anchor at the bottom does the reverse. A midpoint anchor spreads the field in both directions and keeps 1000 meaningful as a reference point rather than as a ceiling.
The reference is held constant across releases within a major version. Moving it silently would invalidate every published comparison, so the conditions under which it moves are stated in the governance rules and prior results are restated rather than quietly replaced when it does.
12.6 Token efficiency
Token consumption is a property of the model rather than the hardware, and it is reported alongside the quality score rather than folded into it.
Two measures publish, because the plain reading and the useful reading are different numbers.
Output tokens per task. Median output tokens the model spends on one sample, regardless of whether it got the answer right. This is verbosity, and it is what most readers mean by token efficiency.
Output tokens per correct task. The first measure divided by the success rate. This is what a correct outcome actually costs, and it is the number that belongs in a budget.
The second is not a restatement of the first. Take two models that each spend 5,000 output tokens on a median sample, so their verbosity is identical. One succeeds every time and the other succeeds half the time. The first costs 5,000 output tokens per correct task. The second costs 10,000, because on average it takes two attempts worth of tokens to arrive at one correct result. Identical on the plain measure, twice the cost on the one that matters.
Hidden reasoning tokens count in both, because an enterprise pays for them in hosted and owned deployments alike.
Neither measure is folded into the quality score. A model that spends five times the tokens to reach the same score is a different proposition to deploy, and combining the two would make a score of 1150 ambiguous between accurate and efficient, and mediocre at both.
12.7 The coding score
The Coding Agent test in chapter 21 is not a persona for the same reason. It runs jobs of its own, so it is reported as a separate measurement and not as a weighting of these five.
13. Performance Methodology
Speed in Signal65 PINNACLE is the usable capacity of a fully loaded platform, joined to the scored workload through a token recipe measured on the same runs. No published figure rests on a wall clock.
13.1 Full box saturation and the concurrency sweep
Speed is measured by loading a box rather than by timing one request. A saturation instrument sweeps concurrency, running N agents against the platform simultaneously and recording per agent decode rate, first token latency, and aggregate throughput at each rung.
The sweep does not stop when aggregate throughput peaks. It continues until a rung fails one of the two service level gates, and it stops as soon as either gate is breached rather than waiting for both. Past the aggregate peak a platform is less throughput efficient per agent, and it is frequently still serving every agent acceptably, so the concurrency it can carry can keep rising after the throughput it can deliver has begun to fall. Stopping at the peak would discard that range.
The search strategy is a coarse scan followed by refinement around each transition. Rungs are spaced widely, then narrowed to locate both the aggregate peak and the last rung that clears the gates. Every rung run is published, including the ones that failed and which gate each one failed on.
Single stream speed is deliberately not the measurement. One lucky fast request tells a buyer nothing about how many capable agents a platform can serve at once, and a productivity number that rewarded it would reward the wrong engineering. Loaded aggregate capacity rewards scalability, which single stream measurement cannot see at all.
13.2 Why no published figure rests on a wall clock
Any duration recorded in any trace is contaminated by the box it ran on, the load that box was under, the network in the path, and thermal state. Durations do not port between environments and they are not reproducible by a third party, which makes them unsuitable as the basis of a published comparison.
What survives is two portable quantities. A token recipe from the scored traces, which is what the intelligent behavior cost in tokens. And rates from the saturation instrument, which is how fast a platform emits tokens under load. These are multiplied at the end. A wall clock is never multiplied by anything.
13.3 The decode bound argument
Carrying a single rate is only legitimate if one phase dominates task time, and on this workload it does. Prefill runs approximately 500 times faster than decode, measured in our testing at roughly 19,000 tokens per second against roughly 40 tokens per second per agent on the largest models in the roster. Prefix reuse across rounds is high, which makes the prefill tail cheaper still.
So the portable number is decode aggregate throughput, applied to the completion token count from the scored trace. The naive alternative of dividing total tokens by one blended rate is wrong by construction, because prefill and decode differ by two orders of magnitude and growing context has to be charged on the uncached delta per round rather than on the full context every time.
Capacity is not one number. The sweep continues past peak throughput, two operating points come out of it, and a rung counts only if it clears both service level gates.
13.4 The gates, and the two operating points
Raw peak aggregate is gameable. A platform can admit an enormous number of agents and slow every one of them to a crawl, and the aggregate number will look excellent while no agent is doing useful work. Two gates close that, and a rung has to clear both to count.
Figure 7: The two service level gates. A rung counts only if it clears both.
Two operating points come out of the sweep and both publish, because they answer different questions.
Figure 8: The two reported operating points. Where they diverge, the platform is trading per agent efficiency for headroom.
The first token ceiling exists for a reason that is only obvious after it has caught something. A stack under pressure can admit sessions into a queue and report healthy decode rates for the subset it is actively serving. Per agent decode is therefore not sufficient on its own. Measuring the latency to the first token across all admitted agents exposes the difference between agents being served and agents being held.
Sixty seconds is deliberately permissive. It is a liveness threshold rather than an interactivity one, because agentic work is dispatched and collected later rather than watched. Under normal operation the gate never binds, and it binds sharply once a stack begins queueing, which is exactly the behavior it exists to catch.
Signal65 Comment: Agent count is a capacity number and a quality of service axis, and it is reported on the silicon lens. It is deliberately not a term in the productivity composite, because concurrency cancels out of tasks per hour. At equal in service aggregate throughput, tasks per hour is identical however many agents share it.
13.5 The bridge to tasks per hour
The rate comes from the box and the token cost comes from the task. They are joined per token.
Every term is either measured or explicitly derived. R is measured on the reference platform under load. C is measured on the scored runs. The 3600 converts seconds to hours. Nothing in the chain is estimated and nothing is a duration.
This construction is also hard to game in the direction vendors are usually accused of gaming. A configuration tuned for throughput at the expense of answer quality still owes the quality term in the composite, and a configuration tuned for a single fast stream never reaches a competitive R because R is measured under load.
13.6 The measurement honesty check
The saturation instrument runs a single task, and the entire speed methodology depends on that task loading a model the way the scored suite does. That is an assumption, and it was tested rather than asserted. The result below includes the imperfect part.
The tokenizer carries the argument. A model uses the same content to token mapping in every instrument, so a saturation task that holds a fixed volume of content rather than a fixed token count loads each model to its own representative point automatically. A token efficient model legitimately sits lighter, which is a real efficiency to credit rather than an artifact to normalize away. The check is whether the saturation context lands where the scored workload actually lives, and it is run on models that appear in both instruments with the same tokenizer.
Figure 9: Saturation context against each model own scored p95 context. Measured. The median scored sample sits at 46K, so the saturation anchor is the loaded normal point rather than the average.
Both land within roughly 10% of the scored p95 for that model, on its own ruler. The cross-model relationship is directionally consistent as well, with the lighter tokenizer sitting at approximately 0.64 of the heavier one in the saturation instrument against 0.74 at scored p95.
A residual remains. Token accumulation across rounds depends on what the agent reads and how it proceeds, both of which vary run to run, so a small mismatch is inherent to measuring agentic work rather than a sign that something is wrong. The cross check covers two tokenizer families, which is a small sample for an assumption this load bearing.
13.7 Publishing the sweep
The full concurrency sweep is published rung by rung, including the levels that failed and which gate each one failed on. A capacity claim of the form this platform sustains N agents is only checkable if a reader can see the rung above N failing, and can see which of the two thresholds it failed.
This also disciplines the claim internally. Publishing the failing rung makes it impossible to quietly report a peak that was never sustainable, and it gives a vendor engineer something specific to argue with, which is a better outcome than a headline number they can only disagree with in general terms.
14. PINNACLE Intelligence Throughput
Quality and speed are reported independently and can be read that way. They are also combined into one figure, because a buyer comparing platforms eventually has to trade them off and pretending otherwise pushes the judgment onto the reader without giving them the tools to make it.
14.1 The composite
At w equal to 0.75 the speed term is a square root. Tripling the usable capacity of a platform raises Intelligence Throughput by a factor of roughly 1.73 rather than 3. That is the weighting doing its job. Scalability is rewarded and raw speed cannot buy quality cheaply.
14.2 The weighting is judgment, and it is owned
Three to one is an editorial judgment by Signal65. It is not derived from data and it is not the output of an optimization. It is stated as judgment everywhere it appears, it is the same weighting for every vendor and every result, and the reasoning behind it comes from advisory work rather than from the benchmark.
Three observations from that work point the same direction. Enterprises abandon agentic pilots over accuracy far more often than over latency, and a pilot that produces wrong output does not get a second round regardless of how fast it produced it. A wrong deliverable also costs more than a slow one, because someone has to find it, which means redoing the work that was delegated in the first place and absorbing whatever the error touched downstream. And latency has a floor below which additional speed stops being worth anything, since work handed to an agent is typically collected later rather than watched in real time, while accuracy has no such ceiling.
Weighting quality three times as heavily as speed reflects that asymmetry. It is a judgment about what enterprises are observed to do rather than a claim about what they should do, and a buyer whose economics differ can recompute the composite from the published inputs.
One weighting publishes, and it is the same one for every vendor and every result. Every input to the composite publishes alongside it, so a buyer whose economics differ can recompute it at whatever ratio their own experience supports.
14.3 Why per task success compounds
The p to the N term is what separates this composite from a throughput number with a quality adjustment bolted on. A workflow of depth N completes only if every step in it completes, so a per step success rate of 0.97 across a depth of five yields roughly 0.86 end to end, and a rate of 0.90 across the same depth yields roughly 0.59. Small differences in per step reliability become large differences in delivered work, and the deeper the workflow the more brutal the arithmetic gets.
This is the honest reason autonomy is hard. It is also why a model that is slightly less accurate and considerably faster does not automatically win, since the compounding term erodes faster than the speed term grows.
Small differences in per step reliability become large differences in delivered work. At depth five, 0.97 per step delivers 0.8587 end to end and 0.90 delivers 0.5905.
14.4 Break even failure cost
Alongside the composite, each model carries a break-even failure cost. It answers a single question. How expensive would one failure have to be before a cheaper, less accurate model stops being the better buy. A model with high accuracy and high token cost has a low break-even, meaning it pays off as soon as failures cost anything meaningful. A fast, cheap, less reliable model has a high break-even, meaning it only makes sense where failures are close to free.
Expressed that way, the number tells a reader something about their own tolerance for autonomy rather than something about the model in isolation, which is what makes it useful for a procurement conversation.
15. Precision and Quality Parity
Reduced precision buys throughput, and the question a buyer needs answered is what it costs in return. Signal65 PINNACLE measures the answer as well as the rate, on the same runs, which turns that question into something we can measure directly.
15.1 Why a cross precision throughput number is not a comparison
Reduced precision produces more tokens per second. That is the point of it. A comparison that reports tokens per second at one precision against tokens per second at another has measured two different workloads and presented the result as though it measured one, and the direction of the error always favors whichever side reduced precision further.
The comparison is only meaningful if the answers are equivalent. Establishing that requires a quality measurement taken on the same configuration, which is exactly what a throughput only benchmark cannot supply.
15.2 The quality parity gate
Matched precision is the published default on the leaderboard. A cross platform capacity comparison is reported at equivalent precision unless the paired quality runs demonstrate that the answers hold, and where a vendor recommends a reduced precision configuration for production, the quality run at that precision is published alongside the capacity number rather than assumed.
The practical shape of the gate is a paired run. The same model, the same scenarios, the same harness, at two precisions, scored the same way. If the scores are within measurement noise, the precision reduction is free and the throughput gain is real. If they are not, the throughput gain has been partly paid for in answer quality and the comparison has to say so.
15.3 Quantized models are separate entries
A quantization changes the weights, so a quantized model is not the same model as the one it was derived from and it is never merged with it. Every quantized configuration appears in published results as its own entry, with its precision in the model identifier.
Coverage here is deliberately narrow at launch. Signal65 has not systematically swept quantizations for every model, so where multiple precisions of one model exist in the results they exist because that pairing was worth testing, not because the roster is comprehensive on this axis.
15.4 Where reduced precision stops holding
Precision effects are not uniform across task length. Reduced precision configurations that hold quality on short tasks do not necessarily hold it on long context work, where small numerical differences accumulate across many rounds and many tokens of context. A parity check run only on short tasks can clear a configuration that will not hold in a deployment.
This is why the parity check runs on the same agentic and long context workloads as the rest of the suite rather than on a short proxy.
15.5 The serving stack is part of the configuration
Model scores in this suite do not change with the accelerator, and we know that because we tested for it. Scores were originally computed separately on each hardware platform. Across MI300X, H200, and B300 the differences in both component benchmarks sat inside run to run noise, so the practice was dropped and quality is now measured once per model. The reference platform for PINNACLE Speed is B300, which puts every model on the same box for its throughput terms. Multiple platforms enter the suite only in the silicon capacity lens, where throughput and sustained agent count are the quantities being compared.
The serving stack is a different matter, and it is the one configuration axis that has visibly moved answer quality in our testing. The mechanism is rarely intentional. Kernel bugs, version regressions, and misapplied optimization flags all surface the same way, as a configuration that runs fast and returns wrong output. Multi token prediction is the clearest example encountered so far, since it is supposed to leave outputs unchanged and simply arrive sooner. In practice, specific model and stack combinations have produced degraded scores and, in some cases, garbage tokens on a fraction of requests while the throughput number looked excellent.
This is why a quality check runs ahead of a capacity sweep rather than alongside it. A configuration that produces wrong output is not a fast configuration, it is a broken one, and no throughput figure derived from it is worth publishing. A vendor recommended flag that reduces answer quality is exactly the finding a buyer following that vendor cookbook needs to see.
Current results are overwhelmingly vLLM. A small number of models have been measured on SGLang as well, and those comparisons publish as a supplementary view rather than as a claim of broad cross stack coverage. Expanding that coverage across the roster is planned for subsequent releases, and until it lands, no result in this suite should be read as a general statement about how a model behaves on a stack it was not measured on.
Hardware support for numeric formats is the remaining axis. Platforms do not offer the same set, so a like for like comparison at a format only one platform supports is unavailable. Where that happens the comparison is reported at the highest format both platforms support, and the reduced precision result publishes separately with its own quality run attached.
15.6 What this means for a cost per token figure
A cost per token won at reduced precision is a cost per token for a different answer distribution. Where the parity check passes, the cost figure stands and the platform deserves the credit. Where it does not, the honest quantity is cost per correct task rather than cost per token, and the two can point in opposite directions.
16. Live Simulation and Gradability
There is a real and interesting comparison to be drawn between measuring recorded work and measuring generated work, and the two are often treated as competitors when they are better understood as instruments with different reach.
A recording carries something a generated scenario cannot, which is the actual shape of production traffic. A generated scenario carries something a recording cannot, which is a correct answer. Both are worth having. Only one of them supports a statement about whether the work came out right.
You cannot grade a recording. A trace contains what happened, not what should have happened.
16.1 What a live agentic session is
An agent receives a job, explores an environment it has not seen, chooses its own path, and produces a set of artifacts. It does not know how many rounds the job should take. It does not know which files matter. It does not know that a second copy of a record exists three directories away, or that the rule it needs sits in the fourth section of a handbook rather than the first.
What it does have is the freedom to be wrong in its own way, and that freedom is the measurement. A model that takes eighty rounds where another takes thirty, or that abandons a job partway, or that reconciles a contradiction incorrectly, has revealed something that no constrained replay would surface.
16.2 Why a recording cannot be graded
No ground truth was ever created for a recorded session. The data was whatever the data was, and no process wrote down the correct outcome before the work started. So there is no answer key inside the trace to check an output against.
This is why benchmarks built on recordings report throughput and cost and never report quality. It is not an oversight in those benchmarks and it is not a criticism of them. It is a property of the artifact they are built on. A replay can tell you exactly what a stack costs to serve a given shape of traffic, which is a genuinely useful thing to know.
16.3 Why measuring correct work requires generating it
The correct answer has to be created in the same moment as the environment. That is the only construction in which a deterministic grader has something to compare against, and it is the reason generation is a requirement rather than a convenience.
Three layers carry the argument.
- The work is inherited. What the job consists of comes from an external occupational taxonomy, so the choice of tasks is not a Signal65 invention.
- The instance is fresh. Every run regenerates, so there is nothing static to memorize or optimize toward.
- The answer is a generated answer key. The correct outcome exists before the agent starts, which is what makes deterministic grading possible at all.
Generation is the mechanism. Gradability is the point. The distinction that matters when comparing instruments is not whether a workload was recorded or generated, but whether the resulting measurement can be checked against a known correct outcome.
16.4 The consequence for capacity measurement
There is a second consequence that matters specifically for platform comparison. A model prevented from choosing its own path cannot show that it would have taken more rounds. A replay fixes the request schedule, so the traffic shape is identical regardless of which model is behind it, and the difference between an efficient model and a wasteful one disappears from the measurement entirely.
Under live execution that difference is preserved. A model that needs more rounds and more tokens to reach the same outcome consumes more of the platform, and the capacity number reflects it. That is the correct behavior, because it is what the platform will actually experience in production.
17. Reading the Results
The full result set publishes alongside this document. The three lenses below carry their own reading rules, with a small number of worked examples chosen to show the shape of an answer rather than to carry a headline.
17.1 Model intelligence
The model lens reports the PINNACLE Model Score, the five persona scores beneath it, token efficiency, and fabrication rate. It is hardware independent, which means it answers the first of the three sequential procurement questions and can be read before any infrastructure decision has been made.
The default presentation is a two axis view with quality on one axis and token efficiency on the other, because the two are independent and a single ranking hides the trade. A model can lead on quality and sit at the bottom on efficiency, and for a high volume deployment that combination is often the wrong buy. Persona selection re-ranks the table, which is where a leaderboard becomes an answer to a specific question rather than a general one.
17.2 Silicon capacity
The silicon lens reports what a platform sustains under load. Usable aggregate capacity at the gates, the concurrency at which it was reached, first token behavior at that rung, and the scaling and collapse profile across the sweep. Configuration is disclosed inline, including serving framework, precision, node shape, and software versions, because a capacity number without its configuration is not a measurement anyone can act on.
The example below shows the shape of a generational within vendor comparison across a model roster, which is the most straightforward reading the lens supports.
Figure 10: Generational capacity comparison within one accelerator vendor. Measured. Usable aggregate is decode tokens per second at the highest concurrency rung clearing both gates, and at N is that concurrency. Gains are derived as the ratio of the two measured aggregates.
Three things in that table drive the reading rules for this lens.
- The gain is not uniform and the spread is a finding. A generational comparison reported as one number would have collapsed a range from 1.12x to 2.61x into an average that describes none of the six cases. In our testing, sparse mixture models and larger models gained most, and the smallest gain landed on a dense mid size model.
- Larger and sparser models gained most. The two mixture of experts entries and the largest dense model post the biggest generational improvements, while the smallest gain lands on a mid size dense model. Architecture predicts how much a generational step is worth to a given deployment more than parameter count does.
- Concurrency moved more than throughput in several rows. Aggregate capacity and the concurrency at which it was reached are different numbers, and a platform that reaches a similar aggregate at three times the agent count is a materially different proposition for a service deployment.
17.3 Solution economics
The solution lens reports cost per correct task and the utilization crossover between renting and owning. Hosted models are priced at their full API bill, which counts input tokens, cached input at the vendor's discounted cache-read rate, and output tokens, all at published list rates. Open models are priced as the share of a rented node-hour the work consumes, at a stated utilization. Both sides count every token the job consumed, and the tokens spent on failed attempts are billed to the tasks that finished, which is what makes the two directly comparable. Pricing hosted models on output tokens alone understates their cost per correct task several times over on agentic work, because the input an agent re-reads on every round dominates the hosted bill, and an earlier draft of the tooling made exactly that error before it was corrected.
The crossover is the point at which amortized owned infrastructure becomes cheaper than per token pricing for a given volume of correct work. It moves with model choice, with token efficiency, and with utilization, and it publishes as a curve so a reader can locate their own volume on it.
17.4 Configuration disclosure
Every published figure carries the configuration that produced it. Model identifier and precision, serving framework and version, node shape and accelerator count, context length, concurrency rung, and any tuning applied along with where that tuning came from. This appears with the result rather than in an appendix, because a reader evaluating a number should not have to leave the number to find out what it means.
17.5 Coding agents
The coding results are published beside the enterprise results on the model and solution lenses, under the name Signal65 PINNACLE Coding Agent. The model lens reports the Coding Agent Score, a Work Breakdown of the two classes of work behind it, each class on its own with its interval, and a table with the share of samples answered right, token use and the Model Score for the same configuration. The solution lens reports a cost frontier against cost per attempt and cost per passed job, for Frontier Class Work and for Local Class Work and for hosted and open-weights models, with hosted cost from the provider's bill and open-weights cost from a rented B300 node. Section 21.9 sets out the reading rules, and the Coding Agent test FAQ answers the common questions.
18. Governance
Benchmark results are only as useful as the process that produced them, and a reader deserves to know that process before they are asked to rely on a number. Signal65 publishes its governance rules in advance for that reason, so that anyone can see how results are produced, refreshed, corrected, and disclosed, and can hold the published record against the stated rules.
Vendor involvement is part of that process rather than something worked around. Vendors know their platforms better than any outside party does, they are frequently the first to spot a misconfiguration, and results are stronger when they have had the opportunity to check the facts. The rules below set out where that involvement helps and where the line sits. The full document publishes alongside this paper.
Figure 11: What the governance document covers.
Two of these carry more weight than the rest. The vendor review window exists because factual errors are worth catching before publication and vendors are frequently the party who can catch them. It carries no approval right, and a vendor disagreement about a result that is factually correct does not delay or alter publication. Funding disclosure is stated per result rather than as a policy line, because a blanket statement that Signal65 works with vendors tells a reader nothing about the specific number in front of them.
The coding results carry one disclosure of their own. Before publication, nine open models were first run with a per-request limit on output tokens that cut off long responses. Those runs were discarded and rerun without the limit, and only the rerun results are published. Grading rules and briefs were also revised during calibration, before any result was published, and the change log lists those revisions as calibration and not as corrections.
Underneath the rules sits the structural point. Scoring is deterministic and the methodology is published, so a result cannot be argued into a better number and any party with the hardware can reproduce it. Rules matter, and rules that rest on a reproducible measurement matter considerably more.
19. Limitations and What We Do Not Claim
Every item below is something a determined critic would otherwise find on their own.
19.1 On coverage and grounding
- The testable set is a Signal65 determination. The US Department of Labor built the taxonomy. The judgment that 26 of the 41 activities are agent testable is ours, reached through the two gates in chapter 6, and no government body reviewed or endorsed it.
- Coverage is breadth of activity type, not depth within an occupation. No score in this suite certifies a model for a named job, and no coverage percentage should be read as a share of any occupation or of the economy.
- Coverage across the 26 is not uniform. Eighteen are exercised directly and eight appear only as accompanying work products.
- Four activities are absent because we cannot score them. Not because agents cannot perform them. The outcome lives in another person response and there is no deterministic key.
- The activities ground task design and are not a scoring dimension. No result is attributed to an individual activity, because the activities overlap and one scenario exercises several at once.
- The suite measures enterprise agentic work only. It does not measure depth of world knowledge, quality of prose, any modality beyond text, conversational range, or progress toward general artificial intelligence. Section 2.5 sets out the boundary in full, and rankings here should be expected to differ from rankings on general capability benchmarks.
- The scenario suite is 7 items. Three as-found, three governed counterparts, one governed only. That is a small suite, sampled deeply at 40 samples per scenario per model, and depth of sampling is not the same thing as breadth of scenario coverage.
19.2 On scoring
- Deterministic scoring cannot grade open ended output. Anything requiring qualitative assessment is out of scope by construction. That is the price of removing model judges and it is a real price.
- Sandboxes are synthetic. They are generated rather than drawn from real enterprise systems, which is the cost of contamination immunity and a generated answer key. The as-found environment is constructed to be realistic and it is still constructed.
- The persona layer is a weighting scheme over shared measurements. Personas do not run distinct scenarios and do not run at distinct context lengths. Every difference between two persona scores for the same model is a difference in weighting, applied to identical underlying measurements.
- Retrieval is measured at one context length. All retrieval, aggregation, and fabrication figures are taken at 128K. Models that hold accuracy well beyond that length and models that collapse just past it are not distinguished by any number in this suite, and the published context work in section 10.6 is the reference for that behavior.
- Fabrication rate is measured on generated absences. The probes test entities that were never generated and fields deliberately left empty. That is a controlled form of the failure. It does not capture fabrication arising from genuine ambiguity in source material, where two documents support different answers and neither is absent.
- We do not claim to predict on the job performance. We claim verified task completion under stated conditions.
19.3 On performance
- The three to one weighting is judgment. A different buyer would reasonably choose differently, and the inputs are published so they can.
- Tasks per hour is derived rather than timed. The transform is stated in section 13.5 and no published figure rests on a wall clock.
- Launch capacity results run one serving framework and one node shape. A different framework or a different node shape can change the ranking, and results should not be read as a general property of the silicon.
- Results reflect software versions at a moment in time. Serving stacks move quickly, and a result is a measurement of a stack on a date rather than a permanent property of a platform.
- The context cross check covers two tokenizer families. That is a small sample for a load bearing assumption, and it is stated in section 13.6 rather than left to be discovered.
19.4 On independence
Signal65 is an independent research and analysis firm, and independence in this business is demonstrated rather than asserted. Ours rests on four things a reader can check directly.
- The methodology is open. Everything required to understand how a number was produced is in this document and the governance rules that accompany it.
- The scoring is deterministic. No result in this suite is a matter of opinion, ours or anyone else. A score cannot be argued into a better number, by a vendor or by us.
- The configuration is disclosed with every result. A reader who doubts a figure knows exactly what produced it and can say precisely what they would change.
- Any party with the hardware can reproduce it. That is the strongest guarantee available in benchmarking, and it is available here because the suite was built to make it available.
Signal65 works commercially with technology vendors, including vendors whose products appear in these results, and that funding relationship is disclosed per result rather than as a blanket statement. Reproducibility is what makes the disclosure meaningful.
19.5 On the coding test
- One task family. Every coding job asks the model to build a PICARD generator in Python. The jobs vary widely in business content and in what has to be built, and they are still one shape of engineering work. Debugging an existing service, reviewing a change and working across languages are not measured.
- The engine is new to every model. The test runs on PICARD v2, which has not been published. Version 1 is public, and a model that has seen it starts with the advantage experience of a related library gives an engineer, and no more, because every attempt is graded on seeds rendered after the attempt ends.
- Attempt counts are modest. Thirty attempts per hosted configuration leaves intervals of about 30 points near the top of the table, and the leading configurations read as a tier.
- Some Local Class results are not yet measured. Hosted frontier configurations without a Local Class run are shown at 100% Local Class Work until measured, which raises their score most when their Frontier Class pass rate is low.
- Open-model cost rests on enterprise-agent throughput. It is a bridge until a coding capacity run replaces it, and open-weights models with no capacity run are not priced at all.
- The structural limits are judgment. The 70% repetition limit and the 2% limit on revealing details are stated thresholds chosen by the test designers, and a different threshold would move some verdicts near the line.
20. Roadmap
The suite described in this paper is the first release rather than the finished instrument, and the work below is where it goes next. Several of these are already underway and all of them are open to input from the partners reading this document, which is a large part of why the roadmap is published at all. Nothing here carries a date and nothing here ships at launch.
20.1 A sub-agent environment
A third environment, governed but much shorter, where a scenario is one or two steps rather than a full workflow. A large share of real deployments are built this way, with an orchestrator decomposing work and calling narrow agents to execute pieces of it, and the current suite does not measure that shape directly.
20.2 Broader coverage
More of the testable set exercised directly rather than through accompanying work products, a broader model roster, and coverage of new models closer to their release date.
20.3 Decomposed workflows
Measuring an orchestrator routing work to smaller models running narrow subtasks, which is a materially different economic proposition from running one large model end to end and is not comparable to it under the current scoring.
20.4 Solution and cluster validation
Validation across the full stack rather than a single node, including the difference between first light and steady state, which is where a meaningful share of deployed capacity is actually lost.
20.5 Cross stack coverage
Systematic measurement of the same models across serving frameworks, so that the effect of the stack on both throughput and answer quality becomes a published axis rather than a supplementary view. Early results already show that the choice of inference stack changes both, and the second of those is the one buyers do not expect.
20.6 Energy
Performance per watt from measured device power rather than from thermal design power, which assumes full utilization and describes a cooling envelope rather than a workload. The instrumentation exists on both major accelerator families and the work is scoped rather than scheduled.
20.7 Distributed and CPU side measurement
Multi node and split inference shapes, and a companion instrument that isolates the CPU contribution to agentic performance. Early proof of concept work shows that the ranking between two CPUs can invert as agent concurrency rises, with peak single thread performance winning while cores are idle and core count and memory headroom deciding once every core is busy. That is a sensitivity curve rather than a single score, and it is the correct shape for the question.
20.8 Coding agents
A harder set of Frontier Class jobs before the leading configurations saturate the current one, more coding task shapes and languages beyond Python, and a capacity measurement with coding agents on GPU platforms, which also replaces the open-model cost basis with a measured one. At the next major version the coding score is designed to join the headline alongside the enterprise measurements.
21. The Coding Agent Test
Coding agents are among the most widely deployed agent workloads in the enterprise, and the agentic suite described in chapter 7 does not measure them. Its jobs call for routine Python and SQL in the course of business work, but none of them asks a model to engineer software inside a codebase it has never seen, which is what a coding agent is hired to do. The Signal65 PINNACLE Coding Agent test measures that work directly. It runs its own jobs, grades them by running what the model built, and reports a separate number, the PINNACLE Coding Agent Score, that sits beside the Model Score and does not enter it.
The test was developed in close partnership with Kamiwaza and is built on PICARD, the same harness that generates and grades the enterprise jobs, so the design principles in chapter 4 carry over unchanged. Scoring is deterministic, every instance is generated fresh, and no model and no human judges the output.
21.1 Why a separate test, and why it is not a persona
The persona layer in chapter 11 reweights five shared measurements and runs no jobs of its own, a point section 11.4 makes deliberately. Coding work cannot be expressed that way, because none of the five measurements observes it. The Coding Agent test therefore stands as its own measurement with its own score, and it is described as a test, not a sixth persona, so that the persona layer keeps the meaning section 11.4 gives it.
It also stays out of the PINNACLE Model Score for this major version. Folding a new measurement into the headline would re-rank every published result, and it would require every model on the enterprise board to be run on the coding jobs before any coding result could appear. A separate score adds information without moving anything already published. The two are designed to combine when the next major version restates the headline, and the coding score is built in the same units as the Model Score so that the combination does not require a new scale.
The separation matters to a buyer because the two rankings do not match. In our testing they agree broadly across the field and part ways near the top, where most buying decisions sit. The two scores sit on different scales, so the comparison is one of position, and only configurations run at the same setting on both are compared. The leading hosted model on coding sits several places lower on enterprise work, a hosted model in the top three on enterprise work sits outside the top fifteen on coding, and an open-weights model from the lower third of the enterprise table sits in the coding top ten, passing every Local Class job and about a third of Frontier Class ones. A model chosen for governed business work is not, on that evidence alone, the right model for an engineering team.
21.2 The job
Every coding job is the same kind of engineering task. The model is given a business brief, the full source and documentation of PICARD v2, a version of the engine that has not been published, and a working environment, and it is asked to build the generator for a new PICARD job. Concretely it writes a Python program that emits a PICARD test configuration, which in turn generates fresh sandbox data for every sample and computes that sample's answer key from the same data. The prompt carries one instruction above the others, that the engine generates the data and the model does not. The documentation alone runs to about 74K tokens.
The task was chosen because it reproduces the position a coding agent is in on its first day inside an enterprise codebase. The engine is large and unfamiliar, the documentation is long, and the important rules live partly in the source. The output has to be exactly what the brief specifies, it has to be correct on inputs nobody has looked at yet, and it has to hold up as a design, not as a set of answers that happen to be right today. Figure 12 maps each demand of the job to the skill it exercises.
Figure 12: What the coding job demands, and the engineering skill behind each demand.
Building a test for a test engine is not circular. The model never sees the grader, the grader never consults a model, and the artifact the model produces is ordinary software, a program that generates structured data and computes answers from it. That the software happens to describe a PICARD job is what makes it gradable, because PICARD can render the program on seeds chosen after the attempt ends and check every result in code.
21.3 Two kinds of job
The test has two parts that share the task, the harness and the grading, and differ in how much the brief says. Frontier Class Work, the open-ended and more difficult half, gives a long business brief that states what to build and leaves the design to the model, the way a real ticket does. Local Class Work, the fully scoped and easier half, gives a short brief and writes every design requirement into it, including which items must vary between samples. The first separates the frontier. The second separates the large share of the field that passes no Frontier Class job, and it is where models sized for a local workstation show what they can do.
Frontier Class Work is the reason the test exists. Anyone who has used a coding agent has seen it produce a program that runs and still misses the point, a design no competent engineer would choose because it ignores how the system is meant to work. Those failures are not the kind a human engineer makes, and they are the ones that cost a team the most time. The open-ended jobs measure whether a model understands the principles behind the engine and applies them without being told, the design judgment and common sense that separate the right program from a working one.
Figure 13: The two kinds of coding job. Both run in the same harness and are graded by the same code.
Both sets cover ten briefs. The Frontier Class briefs are business scenarios drawn from the same kinds of work the enterprise suite covers, and the Local Class briefs climb from one CSV and one number to several folders, a policy document and files the task does not use.
Figure 14: The ten briefs in each set.
One Local Class brief, reproduced in full, shows what "every requirement stated" means in practice. Its last block is the part a Frontier Class brief never states. The data behind it regenerates on every run, so publishing the brief gives a model nothing to memorize.
Benchmark scenario, small. Data: staff.csv with columns name, department, salary; at least 20 rows, regenerated with different content per sample; every department should appear on several rows. Question: "What is the total salary of the "<department>" department?" The department is picked per sample from the departments that actually occur in that sample's CSV. Answer: the agent writes answer.json beside the CSV containing {"total": N} where N is the exact integer sum. Variation requirements: the department named in the question must differ between samples; the content of staff.csv must differ between samples; no deliverable may have the same content in more than 70% of all samples, pooled over many runs.
Figure 15: A Local Class brief, abridged for layout. The Frontier Class version of the same scenario would leave the variation requirements for the model to work out.
21.4 The working environment
Every attempt runs in the same agent harness, a container with a shell, Python and the engine, driven by an open-source coding agent. The model chooses its own path, reads what it decides to read, runs and tests its own code, and decides for itself when the job is done. Credentials for hosted models never enter the container. A proxy outside it holds them and records which model answered every request, so a result cannot be attributed to the wrong model.
Figure 16: The coding agent environment, identical for every model.
21.5 How an attempt is graded
Grading runs in four stages, all of them code. The configuration the model built is rendered under 25 seeds it never saw, producing 200 complete sandboxes on a Frontier Class job and 125 on a Local Class one, each with its own data and the model's own answer key. Every sample is then checked on its own, the samples are pooled and checked together, and the attempt is scored.
Per-sample checks. The answer sheet confirms that the files exist with the exact names and schemas the brief requires, that the pinned lines appear verbatim, that the question says what it must, and that every guarantee in the brief holds as a SQL count over the rendered data. The sheet then recomputes every answer independently from the rendered files and compares it with the model's key. If the two disagree, the model's key is wrong, whatever the reason.
The check catches errors nobody anticipated. In one sample a payment's invoice was authorized for 2,900 with a materiality of 38%, an allowed excess of exactly 1,102, and the payment was 4,002, exactly on the line. The brief defines over-authorized as more than the threshold, so the payment was clean. The model's key used "at least" and flagged it. Four of its 200 samples landed on that boundary, and all four were wrong.
Checks across samples. A configuration can produce fresh values in every sample while keeping the same skeleton, and a skeleton can be memorized. The pooled checks ask questions no single sample can answer. Whether one value of a deliverable appears in more than 70% of samples. Whether the counts and parameters of the situation change from sample to sample instead of being "always exactly one". Whether a surface detail gives the answer away, measured as the share of uncertainty about a row's class that the detail removes beyond chance, with a limit of 2%. And, where the brief asks for a believable mix, whether the pooled rate holds within the brief's range. Local Class Work checks only the variation the brief states.
One Frontier Class configuration sent every planted breach to the same vendor. The values changed in every sample, but that vendor's payments were findings in 982 of 982 cases across all 200 samples. A model trained on leaked samples would learn that anything paid to that vendor is a finding, and the attempt failed.
Pass. An attempt passes only if every sample passes every check. The share of samples answered right is published beside the pass rate and never folded into it, and the distance between the two carries the same kind of signal as the distance between the two scoring modes in section 12.3. In our testing one hosted model answers 97% of Frontier Class samples correctly and passes none of its 30 attempts, because the data it builds never varies in structure.
Three commitments keep the grading fair as well as strict. Every brief has a reference solution that passes all of its samples with margin. Every check traces to the brief or to a principle stated in the engine documentation every model reads, so a sheet stricter than its brief is treated as a bug. And when nearly every model fails the same check, the strongest included, the sheet is assumed wrong until it is shown to be right. Every scoring rule ships with tests, and each test is proven by mutation, breaking the rule on purpose to confirm the test fails.
A generator whose design never changes passes its samples and fails the job, because the next engineer could predict every sample it will ever make.
21.6 Scoring
Each part of the test reports a pass rate, the share of attempts that passed. The two combine into a composite on a 0 to 100 scale, and the composite is indexed to the reference model in the same way as the Model Score.
Composite C = 50 × Local Class pass rate + 50 × Frontier Class pass rate. PINNACLE Coding Agent Score = 1,000 × (100 − C reference) ÷ (100 − C model).
The reference is Gemma-4-31B-it with thinking enabled, the same model that anchors the Model Score in section 12.5. In our testing it passes 21 of 60 Local Class attempts (35%) and none of 30 Frontier Class attempts, for a reference composite of 17.5. A Coding Agent Score of 2,000 therefore means half the weighted failed coding jobs of the reference, which is how the Model Score already reads. Figure 17 works the formula through five cases.
Figure 17: The Coding Agent Score worked through five cases. The last two rows are measured configurations, in our testing.
The scale has two properties to note. A model that fails every job scores 825, not zero, because the reference itself fails most jobs and the index measures failed work relative to it. At the other end the score rises without bound as failed jobs approach zero, which is true of the Model Score as well, and it has no finite value for a build that passes every attempt on both classes of work. Such a build is listed as "clears every job" with both pass rates shown and no index, and it ranks above every indexed build. The rule is stated now, while no model is near it. The leading configurations in our testing pass about two thirds of Frontier Class jobs.
The weighting is an editorial choice. Half and half states that the everyday coding a local assistant does and the open-ended engineering a frontier agent does matter equally to the score. A different reader could weigh them differently, and the two pass rates are published so that anyone can. Two observations support the choice. Of the 49 configurations measured on both parts, 35 pass no Frontier Class job, so a score built on Frontier Class jobs alone would report most local and mid-size models as having no coding value at all, which the Local Class results contradict. And the ordering barely depends on the weights. Across splits from 30/70 to 70/30 the rank correlation with the 50/50 order stays at 0.998 or above, so the weighting sets the spread of the scale and leaves the ranking alone.
Why the score counts whole jobs. The same argument as section 12.2 applies. A configuration that is right on 199 samples and wrong on one has shipped a generator someone has to debug, so only a passed job counts, and the share of samples answered right is published beside the score, never blended into it. A geometric mean of the two pass rates was rejected because it falls to zero for any model with no Frontier Class pass, which is most of the field today.
Hosted models without a Local Class run. Some hosted frontier configurations have Frontier Class results and no Local Class run at publication. They are shown at 100% Local Class Work, the footer of every coding view says so, and the measured result replaces the assumption when the run completes. The assumption rests on the Local Class results in hand. Every configuration passing more than a quarter of Frontier Class jobs has passed 95% or more of Local Class ones, while the older hosted models measured on both parts pass 55% to 93%. It moves a score most for configurations with low Frontier Class pass rates, which are first in line for measured runs.
21.7 Attempts and confidence
Attempt counts differ by configuration. Hosted models run three attempts per Frontier Class brief, 30 in all, most open models run six, and three open models ran 30 per brief to measure run-to-run spread directly. Every Local Class result in the main run uses six attempts per brief. Every pass rate publishes with its 95% interval, computed with the Wilson method and published beside every pass rate, so the precision of a row is visible next to the row.
The intervals are wide enough to change how the top of the table reads. In our testing 21 passes in 30 attempts spans 52% to 83%, 21 in 60 spans 24% to 48%, and 84 in 300 spans 23% to 33%. Configurations whose intervals overlap are presented as one tier. The reference carries the same uncertainty, and because every score is indexed to it, additional attempts on the reference narrow every score at once.
21.8 Token use and cost
Coding jobs are long. A passed Frontier Class attempt runs dozens of rounds, re-reads a context of 100K to 250K tokens on each one, and can generate more than 100K output tokens in the process. Token use is reported per attempt, as rounds, input tokens, output tokens with hidden reasoning tokens counted, and peak context, for each class of work.
Cost follows the rules of section 17.3 so the two cost views can be read side by side. Hosted models are priced at their full API bill at standard list rates, counting fresh input, cache writes where the provider bills them, cache reads at the discounted rate, and output. Open models are priced as a rented 8×B300 node at $70 an hour, the median on-demand price researched October 19th, 2026, for the node-hours an attempt's output tokens take at the throughput the PINNACLE agent harness measured for that model on that node. Cost per attempt is reported for each class of work separately. Cost per passed job is the cost of every attempt in that class divided by the number that passed, so the tokens spent on failed attempts are billed to the jobs that finished. It is published once a build has at least two passed jobs in the class, so that one pass cannot carry the bill for every failed attempt, and builds below that line show their cost per attempt only. The cost frontier plots the Coding Agent Score against cost per attempt and not cost per passed job, because the score already counts failed jobs and dividing the cost by the pass rate as well would count them twice.
The open-model throughput is a bridge at release, and the footer says so. It comes from capacity measurements taken with enterprise agents, and coding agents carry larger contexts per agent, so fewer of them fit on a node at once. A capacity run with coding agents on the reference box replaces it. An open-weights model that has no capacity run at all is scored on the model lens and not priced on the solution lens until one exists, and the footer names it. Every hosted cost comes from provider-reported usage with the fresh, cache-read and cache-write split.
21.9 Reading the coding results
- Read the interval before the rank. Rows whose intervals overlap are one tier.
- Read the two parts together. A model can pass no Frontier Class job and nearly every Local Class one, which describes a capable assistant for well-specified work that does not yet engineer from principles. The Work Breakdown view shows both halves of every score.
- Compare matching configurations. Effort settings change coding results substantially, and any comparison with enterprise results pairs only configurations run at the same setting on both.
- Check the footer. Hosted models shown at 100% Local Class Work without a measured run, open-weights models scored but not yet priced, and the enterprise-agent basis of open-model cost are all stated there.
The Coding Agent test FAQ at pinnacle.signal65.com/coding-agent-faq, answers the questions this chapter raises in plain terms, with worked examples from the published results.
22. What This Benchmark Is For
An enterprise evaluating agentic AI is not buying a model. It is buying an outcome produced repeatedly, correctly, and at a cost it can forecast. That is a more specific question than the one the field usually asks, and it is the question this suite was built to answer.
That focus is what makes the numbers usable. Because the scope is fixed and stated, every number here is checkable. Scoring is code against a generated answer key before the agent started. Environments regenerate, so nothing static exists to memorize. Configuration travels with every published result. Any party with the hardware can reproduce the measurement and disagree with it specifically rather than in general terms.
What comes out of that is a set of numbers a buyer can act on. Whether a model finishes the job on data as they actually hold it. How far it falls when the data is not clean. How often it invents an answer that was never there. How many agents a platform sustains while all of them stay useful. What one correct outcome costs, and at what volume owning the infrastructure beats renting it.
Those answers will sometimes disagree with the general capability leaderboards, and the disagreement is informative rather than embarrassing. A model that finishes enterprise work reliably at a fraction of the token cost is the better buy for that work even when a larger model reasons more impressively about everything else. Signal65 PINNACLE exists to make that case in numbers rather than in argument.
Important Information About this Report
CONTRIBUTORS
Ryan Shrout
President and GM | Signal65
JV Roig
Senior Generative AI Platform Engineer | Kamiwaza
PUBLISHER
Ryan Shrout
President and GM | Signal65
INQUIRIES
Contact us if you would like to discuss this document and Signal65 will respond promptly.
CITATIONS
This paper can be cited by accredited press and analysts, but must be cited in-context, displaying author name, author title, and Signal65. Non-press and non-analysts must receive prior written permission by Signal65 for any citations.
LICENSING
This document, including any supporting materials, is owned by Signal65. This publication may not be reproduced, distributed, or shared in any form without the prior written permission of Signal65.
DISCLOSURES
Signal65 provides research, analysis, advising, and consulting to many high-tech companies, including those mentioned in this paper. No employees at the firm hold any equity positions with any companies cited in this document. Funding status is disclosed per published result under the governance rules described in chapter 18.
ABOUT SIGNAL65
Signal65 is an independent research, analysis, and advisory firm, focused on digital innovation and market-disrupting technologies and trends. Every day our analysts, researchers, and advisors help business leaders from around the world anticipate tectonic shifts in their industries and leverage disruptive innovation to either gain or maintain a competitive advantage in their markets.
For more information, visit signal65.com
or contact research@signal65.com