Code assistant evaluation that grades generated code for correctness, security, and readability. A model that writes code that looks right but fails at runtime is a bug factory with good typing.
We grade outputs. We do not train models, run fine-tuning, or sell experts we cannot verify.
Code assistant evaluation is grading the code a model writes: functions, scripts, refactors, and explanations of code. It is needed by teams building coding assistants, and by any team letting a model write code that ships. Good means the code is correct, safe to run, and written the way your team would write it. This is human evaluation for the one output type where "looks right" is never enough.
Generated code compiles in the demo and breaks in production. These are the patterns behind that gap.
The code passes syntax and even a quick smoke test, then fails on real input: an off-by-one in the loop, a sort that drops equal keys, a retry that never backs off. Logic bugs are the most common failure in generated code, and the hardest to spot in review because the shape of the code looks right.
String-interpolated SQL, unsanitized input passed to a shell, secrets written to logs, auth checks skipped "for simplicity." The code works and the vulnerability ships with it. We grade generated code against common security patterns so these get a P0, not a shrug.
The model calls a function from the library's next version, or one it invented entirely, with confident parameter names. Hallucinated APIs are the code version of fabricated facts, and they only surface when someone runs the code or reads the docs line by line. We check calls against the libraries you name.
Two-hundred-line functions, single-letter variables, no comments where the logic turns tricky, clever one-liners that take an hour to parse. It runs today and costs you every time someone touches it. We grade readability and structure against your team's conventions.
Reviewers trace the logic against the prompt's requirements and your test expectations. We check edge cases the model tends to miss: empty input, the boundary value, the error path. Where you provide tests or expected outputs, we grade against them directly. A sample that looks idiomatic but computes the wrong result is a P1, full stop.
Injection risks, hardcoded credentials, unsafe deserialization, missing auth checks, overly broad permissions. We grade against a checklist of common vulnerability patterns for your stack, and anything that would be a real incident in production is a P0. This is the layer automated checks support best, with human reviewers confirming what the scanner flags.
Naming, structure, comments where they matter, and fit with your style guide or linter config. Code that works but that your team would reject in review gets a P2 with the specific convention cited. Over a batch, this tells you whether the model writes code your team can actually live with.
The unhappy path is where generated code fails most: no check for empty input, exceptions swallowed silently, retries without backoff, resources never closed. The happy path usually works because the model has seen it a thousand times. We grade what the code does when things go wrong, because production is mostly things going wrong. A sample with no error handling where your conventions require it gets flagged, with the missing case named.
You send 20 to 50 generated code samples plus your rubric, or we help you write one. Include your stack, style guide, and tests if you have them.
Trained reviewers grade every sample against the rubric, backed by automated checks for common bug and security patterns.
Sample-level grades, issue patterns, recommended fixes, and a walkthrough call.
We grade the common ones: Python, JavaScript and TypeScript, Java, Go, SQL, and shell scripting, among others. Tell us your stack when you send samples and we match reviewers who know it, and we check API calls against the libraries you name.
Our grading is review-based: reviewers trace logic, check API usage against documentation, and apply automated static checks for common bug and security patterns. We do not execute untrusted code inside your infrastructure. If you provide test cases with expected outputs, we grade against those directly.
Yes. Send your style guide or linter config and we build it into the rubric. Maintainability grading then reflects your team's actual conventions, not a generic idea of clean code.
That is a bigger surface: planning, tool calls, and the code itself. For agentic coding workflows, see our AI agent evaluation page. A pilot can cover both the agent's decisions and the code it produces.
A pilot runs on 20 to 50 samples, spread across the task types your assistant handles: generation, completion, refactoring, explanation. If one task type dominates your usage, we weight the batch toward it.
Start a pilot: send samples, get every one graded in 2 to 3 business days once scope is confirmed with a walkthrough call. Most teams learn more from that first report than from weeks of internal debate about quality.
Send 20 to 50 samples. Get every one graded for correctness, security, and readability.
Not sure where to start? Talk to us and we will point you at the right evaluation.