In this article
- 🔍 What coder_eval is, and what it is not
- 📄 A task is a YAML file, nothing more
- 🎯 skill_triggered: did the skill fire or not?
- ⚖️ Weighted criteria, not a blunt pass/fail
- 🔁 A/B experiments, datasets and a CI gate
- 💡 Why I find it interesting, for developers and for a small business
- ⚠️ What to factor in before trying it
- ✅ Where to start
🔍 What coder_eval is, and what it is not
coder_eval calls itself "the coding agents gym": a reproducible framework to evaluate, benchmark and A/B-test coding agents. Today it supports Claude Code, OpenAI Codex and Google Antigravity (Gemini), with a plugin SPI to add more. The mechanics are easy to describe: it takes a task written in YAML, prepares a sandbox, launches the real agent with its full tool loop, then inspects the files and commands that came out of that loop.
The distinction that matters is with fixed-dataset benchmarks. SWE-bench and SkillsBench score models against a canonical set of problems and produce a leaderboard; coder_eval has no leaderboard and no dataset of its own, it evaluates the tasks you write. It is also different from tools that grade a model's text output: here you are not scoring a string, you are scoring the result of an agent that read, wrote and executed things in a directory.
repo UiPath/coder_eval what evaluates and benchmarks coding agents and their skills agents claude-code · codex · antigravity (Gemini) · plugin SPI how YAML task → sandbox → real agent → weighted 0.0–1.0 criteria highlight skill_triggered: did the skill fire? yes/no + metrics license Apache-2.0 · Python 3.13+ · pip install coder-eval
Official repo · UiPath/coder_eval ↗
📄 A task is a YAML file, nothing more
The unit of work is a YAML file with four things: a prompt, the agent configuration, a sandbox and the success criteria. The required fields are `task_id`, `description`, `initial_prompt` and at least one criterion; `agent` and `sandbox` are optional and, when omitted, resolve from the experiment layer. There are also `tags` for filtering runs, `reference` for a reference solution, and `pre_run` / `post_run` for commands before and after the agent.
The sandbox has two drivers: `tempdir`, the default, which prepares a temporary directory with a Python environment; and `docker`, the container, the only one the documentation considers acceptable for untrusted tasks — and it says so explicitly, `tempdir` is not a security boundary. On the agent side you configure `type`, `model`, `permission_mode` (`acceptEdits` is the recommended one for evaluations) and the list of allowed tools.
task_id: "hello_world"
description: "Create a Python script that prints Hello, World!"
initial_prompt: "Create hello.py that prints 'Hello, World!'"
agent:
type: "claude-code"
permission_mode: "acceptEdits"
allowed_tools: ["Read", "Write", "Bash"]
sandbox:
driver: "tempdir"
python: {}
success_criteria:
- type: "file_exists"
path: "hello.py"
description: "hello.py must be created"
- type: "run_command"
command: "python hello.py"
timeout: 10
description: "Script must execute successfully"Task Definition Guide · full schema and criterion types ↗
🎯 skill_triggered: did the skill fire or not?
This is the criterion that makes the project worth a look. `skill_triggered` is a binary classifier that scans the run traces for two alternative signals: an explicit call to the `Skill` tool whose parameter matches `skill_name` — namespace prefixes are stripped, so `uipath-agents` also catches `uipath-coded-agents:uipath-agents` — or, for an agent that has no `Skill` tool (Codex, for instance), a command that reads the skill's files off disk under `skills/<skill_name>/`.
The observed label is "yes" when either signal appears, "no" otherwise; the expected label is "yes" only when `expected_skill` matches `skill_name`. Scoring is binary: 1.0 when observed and expected agree, 0.0 otherwise. The recommended pattern is to label every dataset row with the skill that should fire — an empty string for negative rows, where the skill must not fire — and stack one `skill_triggered` criterion per skill against the same dataset. Each one produces its own confusion matrix from the same traces, and you can set a threshold on accuracy, F1 or per-label precision and recall.
- type: "skill_triggered"
description: "uipath-agents activation"
skill_name: uipath-agents # the skill to detect
expected_skill: "${row.expected_skill}" # the row's expectation; "" = negative
suite_thresholds:
recall.yes: 0.70
precision.yes: 0.80Documentation for the skill_triggered criterion ↗
⚖️ Weighted criteria, not a blunt pass/fail
Every criterion has a `weight` (default 1.0) and a `pass_threshold` (default 0.9). A task passes when all criteria reach their own threshold; in parallel a weighted score is always computed with the formula `weighted_score = sum(score * weight) / sum(weight)`, which is useful for spotting trends even when the boolean outcome is already known. This is where the approach separates itself from hand-rolled scripts: partial credit exists, and a degradation becomes visible before it turns into a failure.
The criterion types cover a wide range. There are static file checks (`file_exists`, `file_contains`, `file_check` with includes/excludes/regex, `file_matches_regex`, `json_check` with JMESPath assertions and JSON Schema), execution checks (`run_command` with exit code and stdout matching, `command_executed`), comparison against a reference solution (`reference_comparison` with a similarity threshold), and two LLM judges: `llm_judge`, which grades against a text rubric, and `agent_judge`, which is a full secondary agent with mandatory security defaults on sensitive paths.
By eye, the way it's usually done
- Try three prompts and see if it "seems" to fire
- The model changes and you don't notice
- Two skills with similar descriptions steal each other's triggers
- No number to compare from one week to the next
With a skill_triggered criterion
- Positive and negative rows labelled in a dataset
- Confusion matrix with accuracy, precision, recall, F1
- Suite thresholds that make the run exit non-zero
- A history of weighted scores, so trends are readable
The value isn't the tool itself: it's moving from an impression to a repeatable metric.
🔁 A/B experiments, datasets and a CI gate
On top of tasks sits an experiment layer that runs the same tasks across different variants: model vs. model, prompt vs. prompt, tool on vs. tool off, and — the most interesting case for anyone writing skills — skill plugin installed vs. not installed. It is the natural companion to `skill_triggered`: you measure how much the skill actually moves the result, instead of assuming it. With dataset mode you take a single task and fan it out over many rows, turning a few dozen lines of YAML into a serious suite.
The last piece is the CI gate. There is a composite action on the GitHub Marketplace that installs the pinned CLI, runs the tasks, writes a JUnit report, appends the summary to the job summary and fails the step on any failing task or gate. There is also an optional `minimum-task-score`, a strict floor on the weighted score that stacks on top of the CLI's own exit code. The documentation insists on two things: pin the version, because a framework upgrade must not silently move your results; and do not expose secrets to fork pull requests, because tasks execute agent-generated code.
- uses: actions/setup-node@v4 # the claude-code agent needs the CLI…
with: { node-version: '20' }
- run: npm install -g @anthropic-ai/claude-code
- uses: UiPath/coder_eval@v0 # …then run the gate
with:
tasks: tests/tasks/**/*.yaml
model: claude-sonnet-5
minimum-task-score: "0.8"
env: |
ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}coder_eval composite action on the GitHub Marketplace ↗
💡 Why I find it interesting, for developers and for a small business
On the development side the reasoning is that of regression testing, applied to a part of the system that had none until now. A skill, an MCP server, a CLAUDE.md: they are all artefacts that shape an agent's behaviour without anyone ever verifying them automatically. The day the model changes version, the only thing you notice is that "the agent behaves a bit differently". A YAML task with two or three criteria, run on a schedule, turns that feeling into a red line in the pipeline.
On the business side the point is repeatability over time. If a team is introducing agents into its processes — code review, documentation generation, ticket triage — the question that eventually arrives is "how do we know it still works?". Having a runnable task suite is the difference between governed adoption and a series of anecdotes. I'd add that the per-tool-call, token and cost telemetry the framework collects is as useful as the scoring: it makes visible what an agentic flow really costs before you generalise it across the whole team.
- 01Write the skillSKILL.md with the description that decides whether and when it fires
- 02Write 5-10 dataset rowspositive cases where it must fire, negatives where it must not
- 03One YAML taskskill_triggered plus a couple of criteria on the concrete outcome (files, commands)
- 04Local runcoder-eval plan to validate without spending tokens, then run and report
- 05Scheduled gatethe action in CI on every change to the skill, plus a fixed cadence
The entry cost is low: the value shows up when the run becomes periodic, not on the first execution.
⚠️ What to factor in before trying it
Three things need saying clearly. First: the project is very young. Few stars, very few commits on the main branch, a version still below 1.0 — the action itself is referenced as `@v0` pending the stable release. It does have a solid base around it though: a PyPI package, a Marketplace action, a documentation site, an Apache-2.0 licence and public CI. It is material to study and to use on an internal project, not yet something to put at the centre of a critical process.
Second: evaluating costs money. Every run launches a real agent with real credentials — Anthropic, Bedrock or Gemini — and coder_eval does not proxy or supply model access. The `plan` command validates a task without spending tokens, and it is the first habit worth forming. Third: anonymous usage telemetry is on by default and is disabled with `TELEMETRY_ENABLED=false`; the documentation states it never captures prompts, file contents or repository paths, but it is a conscious decision to make before installing it inside a company. Finally, it requires Python 3.13 or newer.
✅ Where to start
The shortest path is the official tutorial: clone the repo, run `uv sync --extra dev`, and run `coder-eval plan tasks/hello_date.yaml` to see validation without consuming anything. Only then `run` and `report`. Anyone who wants to use it on their own project installs the published package — `uv tool install coder-eval` or `pip install coder-eval` — and points the CLI at their own task files, pinning the version if it ends up in a pipeline.
The broader signal I read in this repo, though, goes beyond the tool. Skills and MCP servers are becoming product artefacts: they have a public surface, observable behaviour and regressions. This is exactly the phase in which, in every other area of software, tests appeared. Whether coder_eval or another framework wins matters little: the pattern — describe the expected behaviour in a file, run it in a sandbox, measure it and gate it in CI — is what's worth taking home.
Frequently asked questions about coder_eval
What is coder_eval and what is it for?
It is an Apache-2.0 open-source framework published by UiPath to evaluate and benchmark coding agents and their skills. It runs a real agent (Claude Code, Codex or Antigravity/Gemini) inside a sandbox against tasks written in YAML, then scores the files and commands actually produced using weighted criteria from 0.0 to 1.0. It exists to turn something usually judged by eye into a repeatable metric.
How do I know whether a Claude Code skill actually fired?
With the `skill_triggered` criterion. It is a binary classifier that scans the run traces looking for a Skill tool call with a matching parameter, or — for agents without that tool — a read of the skill's files under `skills/<name>/`. On a dataset-backed task with positive and negative rows it produces accuracy, precision, recall, F1 and a confusion matrix, and you can gate it with suite thresholds.
Is coder_eval a benchmark like SWE-bench?
No, and the difference is substantial. SWE-bench and SkillsBench are fixed datasets that produce a leaderboard of models on canonical problems. coder_eval has neither a leaderboard nor a dataset of its own: it evaluates the tasks, skills and workflows you write, with weighted criteria and thresholds you decide. It can still wrap an external dataset through its bring-your-own-dataset mode.
How much does running an evaluation suite cost?
It depends on the tokens the agent consumes, because every run launches a real agent with your credentials: coder_eval does not proxy or supply model access. The `plan` command validates tasks without spending anything, and it is the right way to iterate on the YAML. The framework also collects per-tool-call, token and cost telemetry, so the bill for each suite is visible in the reports.
Can I use it as a continuous integration gate?
Yes: there is a composite action on the GitHub Marketplace that installs the pinned CLI, runs the tasks, writes a JUnit report and fails the step on any failing task or gate, with an optional `minimum-task-score` as a floor on the weighted score. Two warnings from the docs: pin the version, and do not expose secrets to fork pull requests, because tasks execute agent-generated code.
Let's talk
If this topic is relevant to you, write to me: comparing notes on code and AI is always time well spent.
