Signal65 PINNACLE Coding Agent test, FAQ
Public questions and answers for the release page and press kit
About the test
What is the Signal65 PINNACLE Coding Agent test?
It measures whether an AI model can do the work a coding agent is hired for, building working software inside a codebase it has never seen. Every model gets the same jobs in the same agent harness, and code grades every attempt by running what the model built. The result is the PINNACLE Coding Agent Score, published beside the PINNACLE Model Score and scored separately from it.
What does a model do in the test?
It receives a business brief, the full source and documentation of an unpublished version of the PICARD engine, and a container with a shell and Python. It has to write a program that generates a new PICARD job, meaning fresh sandbox data for every sample and the correct answer for each one. The model reads what it chooses, runs and tests its own code, and decides for itself when the job is done.
Why that task, and not fixing a bug or building an app?
Because it reproduces the first weeks of an engineer inside an enterprise codebase, a large unfamiliar library, long documentation, an exact specification and output that has to be right on inputs nobody has checked yet. It is also fully gradable. PICARD can run the program on new seeds after the attempt ends and check every result in code, which a bug fix or an app rarely allows without a human reviewer.
What is the difference between Local Class Work and Frontier Class Work?
Local Class Work is the fully scoped, easier half of the test. Its short briefs spell out every requirement, including what must vary from one sample to the next, and they test whether a model can follow a small spec exactly, which is the everyday work of a local coding assistant.
Frontier Class Work is the open-ended, more difficult half. Its longer business briefs say what to build and leave the design to the model, and they test whether it brings the judgment a competent engineer supplies without being told, such as making sure the data it generates cannot be memorized.
Who grades the work?
Code does. No model judges another model and no human rates the output. The grader renders the model's program on seeds it never saw, checks every file, schema and answer, recomputes every answer independently, and then checks the design across all the samples together. Re-grading an attempt gives the same result every time.
What does it take to pass a job?
Every generated sample has to pass every check. On a Frontier Class job that is 200 samples, and one wrong answer or one design flaw fails the attempt. The same rule applies to Local Class jobs, which render 125 samples per attempt. Those easier jobs vary less, so fewer samples judge them reliably, and the stricter check is kept for Frontier Class Work, where design judgment is what is being measured.
How can a model write working code and still fail?
A program can produce correct answers from a design that never changes, for example a count that is always the same or a vendor that is always the one at fault. That passes every individual sample and fails as software, because anyone who has seen a few samples can predict the rest. In our testing GPT-5.6-sol answers 97% of Frontier Class samples correctly and passes none of its 30 Frontier Class jobs for exactly this reason. The share of samples answered right is published beside the pass rate on every view.
Can a model memorize the test or train against it?
Not usefully. Every attempt is graded on seeds rendered after the attempt ends, so there is no fixed answer to learn. The engine the models work in, PICARD v2, has not been published, so every model meets it the way a new agent meets a company's internal code for the first time. An earlier version is public, and knowing it helps only the way experience with a related library helps an engineer.
Which languages and tools does it cover?
Python, SQL, YAML, JSON and the shell, applied inside an unfamiliar engine. It does not yet measure other languages, debugging an existing service or reviewing someone else's change, and those are on the roadmap.
Scoring
How is the PINNACLE Coding Agent Score calculated?
Each part of the test reports a pass rate. The composite is 50 × the Local Class pass rate plus 50 × the Frontier Class pass rate, on a 0 to 100 scale. The score indexes that composite to the reference model, Gemma-4-31B-it with thinking enabled, at 1,000.
The index reads as a ratio of failed work. A score of 2,000 means half the weighted failed coding jobs of the reference, and 4,000 means a quarter. The composite and both pass rates are published with every score.
Why is the reference model Gemma-4-31B, and why 1,000?
It is the same reference that anchors the PINNACLE Model Score, so both scores read the same way. It is a widely used open model from a major lab, and in our testing it passes 35% of Local Class jobs and none of the Frontier Class ones, which puts it inside the range of the field, not at either end.
Why does a model that fails every job score 825 and not zero?
The score measures failed work relative to the reference, and the reference itself fails most jobs. A model that fails every job has about 1.2x the weighted failures of the reference, which the index reports as 825. Scores below 1,000 mean more failed coding jobs than the reference, and scores above it mean fewer.
Is there a ceiling on the score?
The index has no finite value for a build that passes every attempt on both classes of work, because it describes failed work and such a build has none. A build in that position is listed as "clears every job" with both pass rates shown and no index, and it ranks above every indexed build. No model is near that today. The top configuration in our testing passes 70% of Frontier Class jobs.
Why weight the two kinds of job equally?
It states that everyday coding and open-ended engineering matter equally, and it is an editorial choice we own. Most models in our testing pass no Frontier Class job, so a score built on Frontier Class jobs alone would report capable local models as having no coding value, which their Local Class results contradict. The ranking barely moves under other weightings. Across splits from 30/70 to 70/30 the rank correlation with the 50/50 order stays at 0.998 or above.
Is the coding score part of the PINNACLE Model Score?
No. It is published beside the Model Score and does not change it, so every published enterprise result stands as it was. It is built in the same units so that the two can combine at the next major version of PINNACLE.
Why is it called a test and not a persona?
PINNACLE personas reweight the same five enterprise measurements and run no jobs of their own. The coding work runs its own jobs and is graded on its own, so it is reported as a separate test.
Some hosted models show 100% on Local Class Work. Did they all pass every job?
Not yet measured. A number of hosted frontier models have Frontier Class results and no Local Class run at release, and they are shown at 100% Local Class Work until the run completes, which the footer of every coding view states. The assumption rests on the Local Class results in hand. Every configuration that passes more than a quarter of Frontier Class jobs has passed 95% or more of Local Class ones, while the older hosted models measured on both parts pass 55% to 93%. The measured result replaces the assumption when it lands.
Results
Why do some models have more attempts than others?
Hosted models run 30 Frontier Class attempts, most open models run 60, and three open models ran 300 to measure run-to-run spread. Every Local Class result in the main run uses 60 attempts. Pass rates are comparable across those counts, and the precision of each is shown in its interval.
How precise are the results?
Every pass rate publishes with its 95% interval. In our testing 21 passes in 30 attempts spans 52% to 83%, and 84 passes in 300 spans 23% to 33%. Rows whose intervals overlap read as one tier.
Why do effort settings appear on every row?
Effort changes coding results substantially. In our testing Gemini-4-Argon passes 67% of Frontier Class jobs at high effort and 43% at medium. Every row carries its setting in the same form as the enterprise results, and comparisons pair matching settings only.
Why does a model rank differently on coding than on enterprise work?
They measure different work. The enterprise jobs reward reading policy, applying rules and delivering a full set of business outputs on messy data. The coding jobs reward engineering inside an unfamiliar codebase. In our testing the two rankings agree broadly and part ways near the top, so a model chosen for one is not automatically the right choice for the other. The coding results carry each build's Model Score beside its Coding Agent Score for the comparison.
Why are some models missing, or on one view and not the other?
A model appears on the Models lens once it has been run on Frontier Class Work at a published configuration. Preview builds and superseded configurations stay on the coding results with a data note. A model appears on the Solutions lens once it can be priced, which for an open-weights model means the PINNACLE agent harness has measured its throughput on the B300 node. Models that are scored but not yet priced are named in the Solutions footer, and a hosted model with no Local Class run appears on the Frontier Class cost view only. New models are added as they are run, and requests can go through the contact form.
Cost
How are hosted models priced?
At their full API bill at standard list rates, counting fresh input, cache writes where the provider bills them, cache reads at the discounted rate, and output including reasoning. Every token counts, including the ones spent on failed attempts. Gemini 4 Argon is priced at the standard rate, not the promotional one.
How are open models priced?
As a rented 8×B300 node at $70 an hour, the median on-demand price researched October 19th, 2026, for the node-hours an attempt's output tokens take at the throughput the PINNACLE agent harness measured for that model on that node. At release that throughput comes from capacity runs with enterprise agents, which the footer states, until a capacity run with coding agents replaces it. Node utilization on the view scales the open-weights cost only.
What is the difference between cost per attempt and cost per passed job?
Cost per attempt is the average bill for one try at one coding job in the selected class of work, whether it passed or failed. Cost per passed job is the total bill across every attempt divided by the jobs that passed, so the failed tries are paid for by the successes. In our testing GPT-6.1-Sol costs $0.76 a Frontier Class attempt and passes 57% of the time, so a passed job costs $1.35.
The cost frontier plots the Coding Agent Score against cost per attempt, because the score already reflects how often a model fails and cost per passed job would count those failures twice. A cost per passed job is published once a build has at least two passed jobs in that class, so that a single pass cannot carry the bill for every failed attempt. Builds below that line show their cost per attempt and are named under the view.
Can I price Local Class Work on its own?
Yes. Cost per attempt and cost per passed job are published for Local Class Work as well as Frontier Class Work, from the Local Class attempts themselves. The roster differs between the two classes, because a hosted model with no Local Class run cannot be priced on it, and the footer names the builds that are off the Local Class results for that reason.
Can I compare only closed models or only open weight models on cost?
Yes. The cost results can be read for open models on the B300 node alone or for API models alone, with the frontier line recomputed for whichever group is shown, and a price ceiling picks out the most capable build under a given cost per attempt.
Process
Were any results rerun or changed before publication?
Yes, and both are disclosed. Nine open models were first run with a per-request limit on output tokens that cut off long responses. Those runs were discarded and rerun without the limit, and only the rerun results are published. Grading rules and briefs were also revised during calibration, before any result was published, so the published version is the first version of the test.
Can a vendor change or block a result?
No. Vendors may check coding results for factual accuracy before publication, as with every PINNACLE result, and they have no approval right and no veto.
How often will the coding results update, and what comes next?
New models are added as they are run, under the same frozen test version. Planned work includes a harder set of Frontier Class jobs before the leading models saturate the current one, more task shapes and languages, and coding agent capacity measured on GPU platforms, which also replaces the open-model cost basis with a measured one.
Who built the test?
Signal65 developed the Coding Agent test in close partnership with Kamiwaza, on the PICARD harness that underpins the rest of PINNACLE. Signal65 owns the methodology, the governance and publication.