Introducing the Signal65 PINNACLE Coding Agent Test
Signal65 PINNACLE adds a Coding Agent test that drops a model into a codebase it has never seen, grades the design judgment behind the software it builds, and prices what coding work costs.
Signal65 · Ryan Shrout
October 29th, 2026
First major upgrade to Signal65 PINNACLE
Signal65 PINNACLE launched in August as a benchmark for enterprise agentic AI, built to answer a question enterprises were not getting from vendor benchmarks, which is how much correct business work a model, a GPU platform or a full solution produces. It measures that across three lenses. The PINNACLE Model Score grades a model on agentic jobs that read like real business assignments, a workforce review, a fraud audit, a product defect investigation, run against messy generated data and scored only on whether the finished work is right. The PINNACLE Capacity results measure how many of those agents a GPU platform sustains at a service floor. The Solutions results put the two together as the cost of correct work, so a buyer can see what a correct task costs on a hosted API or on a rented node. Five personas reweight the same measurements for different kinds of enterprise users, and every result is graded deterministically in code with no model judging another model.
What we are launching today is a separate measurement. Coding agents are one of the most widely deployed agentic workloads in the enterprise, and none of the PINNACLE enterprise jobs asks a model to engineer software inside a codebase it has never seen, which is what a coding agent is asked to do. The Signal65 PINNACLE Coding Agent test runs its own jobs, grades them by running the software the model builds, and reports its own number, the PINNACLE Coding Agent Score, which publishes beside the Model Score on pinnacle.signal65.com starting today. Coding capability and enterprise business capability turn out to be different things, as the results below show, and a buyer deciding which model goes to an engineering team needs the coding number on its own before the two combine at the next major version of PINNACLE.
Where coding agents still fall short
Coding agents are among the most useful tools to reach software teams in years, and they are where application development is going. A good one will take a description of a feature, find its way around a large codebase, write the code, write the tests and hand back a pull request in the time it used to take to read the ticket. The gaps are real too, and anyone who has handed one serious work has run into the same one. Picture an agent asked to add a vendor payment audit to a finance system, a report of every payment that exceeded its authorized amount by more than the company's tolerance. The code comes back clean. It runs, the tests it wrote for itself pass, and the sample output looks right. Then someone notices that the report flags the same vendor in every test case, because the agent built its test data so that one vendor is always the one at fault, or that it compared amounts with "at least" where the policy says "more than", or that every audit it generates has exactly one finding because the sample it was shown had one. The program is correct on the data it saw and wrong as a piece of software, because the agent never worked out how the system is meant to behave and built the simplest thing that satisfied the examples in front of it.
Those are not the mistakes a human engineer makes, and they are the expensive ones, because somebody has to find them after the code looks finished. Benchmarks that count passing unit tests do not see this failure at all. The Signal65 PINNACLE Coding Agent test is built to catch it, by grading the design of what the model builds and not only its answers, in our testing.
How PINNACLE tests, and what the coding test asks
Every PINNACLE enterprise job runs on PICARD, the engine behind every PINNACLE job, which generates and grades agentic work. For each job PICARD renders a fresh business scenario, the files, the databases, the policy documents and the clutter, from a seed, and it computes the correct answer for that scenario from the same data at the same time. An agent then works the job in a sandbox with the tools it needs, and PICARD grades the finished deliverable against the answer it already knows. Because every scenario is generated fresh and every answer is computed rather than judged, the grading is deterministic, and nothing in the test can be memorized.
The Signal65 PINNACLE Coding Agent test turns that engine into the codebase the model has to learn. Instead of working a job, the model is asked to build one. It receives a business brief, the full source and documentation of an unpublished version of PICARD, and a working environment, and it has to write the program that generates a new PICARD job, with fresh data for every scenario and the correct answer for each. The table below lists what the model gets and what it has to deliver.
| What the model receives | What it has to deliver |
|---|---|
| A business brief, written the way an engineering request is written | A Python program that generates a new PICARD job |
| The full source and documentation of PICARD v2, a version that has not been published, about 74K tokens of documentation with rules that live partly in the source | Fresh, realistic data for every sample the job renders |
| A container with a shell, Python and the engine, driven by an open-source coding agent | The correct answer for every sample, computed from that sample's data |
| 7,200 seconds per attempt, no limit on how much it generates, and the freedom to read, run and test whatever it chooses | A design that holds up across samples, so the next engineer could not predict the data from a few examples |
A job is one brief, and one attempt at it is one run of the model in the harness, from reading the brief to deciding it is done. A sample is one complete scenario the model's program renders, its own files, data and answer key, and a Frontier Class job renders 200 of them for grading. The model never sees those samples, because they are rendered from seeds chosen after the attempt ends. A job passes only when every sample passes every check.
Grading is all code, which continues the deterministic direction every Signal65 PINNACLE test is built on. Files, schemas and pinned lines have to match the brief exactly, every guarantee in the brief has to hold as a count over the rendered data, and every answer is recomputed independently and compared with the model's own key. Then the 200 samples are pooled and checked together, which is the part a unit test cannot do. No model grades another model and no human rates the output, so regrading an attempt gives the same answer every time.
Right answers are not enough
The checks across samples are the ones that catch the failure described above. A generator can produce correct answers from a design that never changes, a count that is always the same, a vendor that is always the one at fault, and it passes every individual sample while failing as software, because anyone who has seen a few samples can predict the rest. In our testing GPT-5.6-sol answers 97% of its Frontier Class samples correctly and passes none of its 30 Frontier Class jobs for exactly this reason. The share of samples answered right is published beside the pass rate on every view, and the gap between the two is the design judgment that Signal65 PINNACLE grades and other coding benchmarks do not.
Figure 1. The Frontier Class work gap, samples answered right against jobs passed, for every build that answers at least half of its open-ended samples correctly.
The per-sample checks catch things nobody anticipated. In one vendor payment audit sample, an invoice had been authorized for $2,900, and the brief set a materiality threshold of 38%, meaning a payment could run up to 38% over its authorized amount, or $1,102 more, before it counted as a finding. The payment in that sample was exactly $4,002, which is $2,900 plus $1,102, right on the line. The brief defines over-authorized as more than the threshold, so that payment was clean, and the model's answer key had used "at least" and flagged it. Four of its 200 samples landed on that boundary and all four were wrong, so the job failed. A competent engineer reads "more than" and "at least" as different instructions, and the Signal65 PINNACLE Coding Agent test holds a model to the same standard.
Two classes of work, one score
The Signal65 PINNACLE Coding Agent test is built in two pieces. They have the same goal, which is to find out whether a model can build correct software in an unfamiliar codebase, and they run in the same harness and are graded by the same code, but they differ in how much of the design the brief hands the model. Frontier Class Work is the open-ended, more difficult half, where a long business brief states what to build and leaves the design to the model the way a real engineering request does. Local Class Work is the fully scoped, easier half, where a short brief writes every requirement out, including what must vary from one sample to the next, and tests whether a model can follow a small spec exactly in an unfamiliar codebase, which is the everyday work of a local coding assistant. Both halves cover ten briefs each, and a job passes only when every generated sample passes every check.
The PINNACLE Coding Agent Score weights the two halves equally and indexes the result to the same reference model that anchors the PINNACLE Model Score, Gemma-4-31B-it with thinking enabled, at 1,000. A score of 2,000 means half the weighted failed coding jobs of the reference. The score is reported separately from the general model intelligence score for now, as we work to update the PINNACLE Enterprise benchmark in the coming weeks, so every enterprise result stands as it was. We chose to weight the two halves equally. Of the 49 configurations measured on both halves, 35 pass no Frontier Class job, so a score built on the open-ended half alone would report capable local models as having no coding value, which their Local Class results contradict. The weighting also barely moves the ranking. Across splits from 30/70 to 70/30 the rank correlation with the 50/50 order stays at 0.998 or above.
Figure 2. Both Coding Agent work classes behind the score, the Local Class half and the Frontier Class half of each of the 22 highest scores. Hosted frontier models without a Local Class run yet are shown at 100% Local Class Work until their runs land.
What models finish the job today
The top of the Frontier Class results reads as a tier rather than a ranking, because 30 attempts per hosted configuration leaves intervals about 30 points wide. In our testing Claude-Opus-5.5 at medium effort passes 70% of open-ended jobs, Gemini-4-Argon at high effort 67%, GPT-6-Astra 63% and GPT-6.1-Sol 57%, and the four intervals overlap. Below them Gemini-4-Argon at medium effort passes 43% and Claude-Sonnet-5.5 37%.
The open-weights leaders are closer to the frontier than their enterprise results would suggest. GLM-5.3 at low effort passes 35% of Frontier Class jobs and DeepSeek-V4.1-Flash at max effort 32%, and each of them passes every Local Class job. Below that line most of the field passes no open-ended job at all, which is the reason the Local Class half exists, and it is where models sized for a PC and workstation show what they can do.
Figure 3. Ranking the best models for Frontier Class work, every configuration with at least one passed open-ended job, with its 95% interval.
One model family across three generations on the same ten briefs shows what the test measures more clearly than any single result. GPT-5.6 Sol answers 97% of samples right and passes 0% of jobs, GPT-6 Sol answers 90% right and passes 7%, and GPT-6.1 Sol answers 100% right and passes 57%. The share of right answers barely moves across the three generations, while the share of jobs whose design holds up across every sample climbs from nothing to more than half. Our read is that the arithmetic of a single sample, reading a record and computing an answer, was solved a generation ago, and what the later generations gained is the ability to read how a system is meant to work out of its documentation and source and carry that understanding through a long, multi-step build without being told, which is the capability that agentic post-training targets and the one a coding agent is built for.
Figure 4. GPT Sol models improving coding capability generationally, on the same open-ended briefs.
The best enterprise agent is not necessarily the best coding agent
A buyer who picked a model on enterprise results and is about to hand it to an engineering team is making an assumption that the capability carries over. The reason the coding score is published separately is that in our testing it does not, and the two rankings part ways at the top of the field, among the models an enterprise is most likely to be choosing between.
The two scores are built in the same units and indexed to the same reference, and they are measured on different work and kept separate on purpose. Coding is reported as a test and not as a sixth persona, because the five PINNACLE personas reweight the same five enterprise measurements and don't run specific jobs of their own, while the coding work does, and is graded on its own. Only configurations run at the same effort setting on both tests are paired.
Across the 57 matched configurations the two rankings agree broadly, with a rank correlation of 0.88, and they disagree at the top. Claude-Opus-5.5 at medium effort is fifth on enterprise work and first on coding. Gemini-3.8-Flash at medium effort is third on enterprise work and seventeenth on coding, passing 3% of open-ended jobs. GLM-5.3 at low effort is thirty-first on enterprise work and seventh on coding, passing every Local Class job and 35% of Frontier Class jobs.
Figure 5. Enterprise rank against coding rank for every matched configuration, with the cases that move furthest named.
They part because the work is different. The enterprise jobs reward reading policy, applying rules and delivering a full set of business outputs on messy data, and the coding jobs reward engineering from the principles of an unfamiliar system. Effort settings pull the two apart as well. Gemini-4-Argon gains most of its open-ended passes between medium and high effort, 43% to 67%, while its Model Score moves by a tenth, and GLM-5.3 passes more Frontier Class jobs at low effort than at max while scoring higher on enterprise work at max. A comparison across the two tests holds only at matching settings, which is why every row carries its effort label in the same form as the enterprise results.
Figure 6. The same model at two effort settings, on the coding test and on the enterprise test.
The practical reading is to pick per workload. Read intervals as tiers rather than places, and use the Model Score column on the coding table to see both numbers for one configuration before deciding which model goes to which team.
Putting a price on agentic coding work
A coding job has two prices. Cost per attempt is the bill for one try, passed or failed, and cost per passed job divides the total bill across every attempt by the jobs that passed, so the failed tries are paid for by the successes. In our testing GPT-6.1-Sol costs $0.76 per Frontier Class attempt and passes 57% of the time, so one passed job costs $1.35. The cost frontier plots the Coding Agent Score against cost per attempt, because the score already reflects how often a model fails and dividing the bill by the pass rate as well would count those failures twice. Cost per passed job is published once a build has at least two passed jobs in a class, so that one lucky pass cannot carry the bill for every failed attempt.
| Build | Cost per Frontier Class attempt | Frontier Class jobs passed | Cost per passed job |
|---|---|---|---|
| GPT-6.1-Sol thinking:medium | $0.76 | 17 of 30 (57%) | $1.35 |
| GLM-5.3 thinking:low (rented B300 node) | $2.10 | 21 of 60 (35%) | $5.99 |
| Claude-Sonnet-5.5 thinking:high | $2.34 | 11 of 30 (37%) | $6.37 |
| Claude-Opus-5.5 thinking:medium | $5.46 | 21 of 30 (70%) | $7.80 |
| GPT-6-Astra thinking:medium | $5.47 | 19 of 30 (63%) | $8.64 |
| Gemini-4-Argon thinking:high | $7.66 | 20 of 30 (67%) | $11.49 |
| Muse-Wonder-1.0 thinking | $14.75 | 6 of 30 (20%) | $73.76 |
Figure 7. The Signal65 PINNACLE Coding Agent Pareto curve for Frontier Class Work, every priced build by its score and the price of one try.
Hosted models are priced at their full API bill at standard list rates, counting fresh input, cache writes where the provider bills them, cache reads at the discounted rate and every output token including reasoning. On that basis a passed Frontier Class job costs $1.35 on GPT-6.1-Sol, $6.37 on Claude-Sonnet-5.5, $7.80 on Claude-Opus-5.5, $8.64 on GPT-6-Astra, $11.49 on Gemini-4-Argon at high effort and $73.76 on Muse-Wonder-1.0. Coding sessions are long, a passed open-ended attempt on Opus runs about 50 rounds, re-reads a context that peaks near 300K tokens and bills more than 10 million input tokens, and the reason the bill is not larger is that 97% of that input is served from cache. The cache-read share is what makes long coding sessions cheaper than their token counts suggest, and it is published for every hosted row.
Figure 8. Cost per passed Frontier Class job for every build with at least two passes.
Open-weights models are priced at a cost per node-hour for a rented 8×B300 node, a rate we research and update continuously on the results views, and at the time of publication it was $78 an hour. An attempt's bill is the node-hours its output tokens take at the throughput the PINNACLE agent harness measured for that model on that node. On that basis GLM-5.3 at low effort costs $5.99 per passed Frontier Class job and Qwen3.8-Flash-Next $8.21, which puts the leading open models inside the range of the hosted frontier rather than below it. That throughput comes from capacity runs with enterprise agents, and coding agents carry larger contexts per agent, so fewer of them fit on a node at once, and the results views state that basis wherever the open-weights cost appears.
Local Class Work changes the picture, and it is the view for everyday scoped work on local-size models. GPT-6-Luna passes 93% of Local Class jobs at $0.06 each, Laguna-XS-2.1 is $0.10 a passed job at a 27% pass rate, and Qwen3.6-27B passes 57% of Local Class jobs at $0.28. GLM-5.3 at low effort passes every Local Class job at $0.64 each on the rented node. Read the pass rate beside the price, because a model that is cheap per passed job and passes a quarter of the time is a different tool from one that passes nine times in ten.
Figure 9. Cost per passed Local Class job, the 16 cheapest among builds with at least two passes.
Some results are scored and not yet priced. DeepSeek-V4.1-Flash, GLM-5.3-Flash, the Qwen3.8-2.4T builds and several smaller open models have no capacity run on the B300 node yet, so they appear on the Models results and not on the cost results until one exists. Kimi-K3 has one passed Frontier Class job, which is too few to price a passed job. The Solutions footer names every build in that position.
The Local Class results also point at a question we intend to answer soon. The open models that pass fully scoped coding jobs for well under a dollar are the size that runs on a developer's own machine, on an NVIDIA DGX Spark or DGX Station, an RTX Spark or an AMD Strix Halo system, where the cost of a passed job is the hardware already on the desk and the electricity it draws rather than a token bill or a node-hour. A development team that routes its scoped, well-specified work to a local model and reserves the hosted frontier for the open-ended engineering changes its cost structure. We are using this class of workload for exactly that purpose, and PINNACLE for client systems, measuring these models on the hardware they run on locally, is coming soon.
Where it goes
The coding results publish today as a second results section on the Models and Solutions lenses at pinnacle.signal65.com, with the PINNACLE Coding Agent Score, the Work Breakdown and the two classes of work on one, and the cost frontier and cost per passed job for both classes on the other. The methodology page, a new chapter 21 in Measuring Correct Work v1.1 and a Coding Agent test FAQ carry the full description. The test was developed in close partnership with Kamiwaza on the PICARD harness that underpins the enterprise tests and the rest of PINNACLE. New models are added as they are run under the same frozen test version, and a harder set of open-ended jobs is already planned for the day the leading models saturate this one.
The Signal65 PINNACLE Coding Agent test gives enterprises something they have not had: a clear, measured view of what coding work costs on each model and each kind of infrastructure, with the failed attempts in the bill and the design of the finished software graded. It shows which models can engineer from the principles of a system they have never seen and which can only follow a spec, it prices both kinds of work on a hosted API and on a rented node, and it separates the models that belong on a frontier API from the ones that belong on a developer's desk. That is the information an enterprise needs to decide where its coding agents should run, and Signal65 intends to use it, with vendors and with buyers, to define which infrastructure is most valuable for each coding capability as the models and the hardware keep moving.