CompletionKit

com.completionkit/evalsVersion 1.0.0

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

Status

Responds475 msNov 14, 2026, 13:52 UTCAnswered our handshake without sign-in.

Registry data may be out of date

We could not refresh it, so this page shows the last copy.

Last checked: Nov 14, 2026, 13:52 UTC
Latency: 475 ms
Remotes responding: 1 of 1
Sign-in: API key
Hosting: Hosted outside the EU (detected)
Registry data as of: Nov 14, 2026, 21:12 UTC

Data from the official MCP Registry, checked by run.zero. How we check

Connect

Streamable HTTP

https://completionkit.com/mcpResponds475 msNov 14, 2026, 13:52 UTC

No sign-in

Claude Code

claude mcp add --transport http evals https://completionkit.com/mcp --header 'Authorization: Bearer <YOUR_TOKEN>'

Replace <YOUR_TOKEN> before running.

Claude Desktop and claude.ai

Settings > Connectors > Add custom connector
Name: evals
Remote MCP server URL: https://completionkit.com/mcp
This server expects the header Authorization.

Cursor

{
  "mcpServers": {
    "evals": {
      "url": "https://completionkit.com/mcp",
      "headers": {
        "Authorization": "Bearer <YOUR_TOKEN>"
      }
    }
  }
}

Replace <YOUR_TOKEN> before running.

VS Code

code --add-mcp '{"name":"evals","type":"http","url":"https://completionkit.com/mcp","headers":{"Authorization":"Bearer <YOUR_TOKEN>"}}'

Replace <YOUR_TOKEN> before running.

Built your own server? run.zero hosts MCP servers on European infrastructure.

Host your own MCP server on run.zero

Tools

54 tools

Tools of CompletionKit
	Description		Read-only	Idempotent	Open world
`agreements_create`Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.	Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'.	Not declared	Not declared	Not declared	Not declared
`agreements_list`List agreements. Filter by run_id, response_id, metric_id, or created_by.	List agreements. Filter by run_id, response_id, metric_id, or created_by.	Not declared	Not declared	Not declared	Not declared
`datasets_create`Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.	Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and "actual_output" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Every column is also available to the prompt as a variable.	Not declared	Not declared	Not declared	Not declared
`datasets_create_from_url`Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.	Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked and the download is capped at 10MB. First row is the header; the "expected_output" (answer key) and "actual_output" (pre-made output) columns are recognized specially, overridable per run.	Not declared	Not declared	Not declared	Not declared
`datasets_delete`Delete a dataset	Delete a dataset	Not declared	Not declared	Not declared	Not declared
`datasets_get`Get a dataset by ID	Get a dataset by ID	Not declared	Not declared	Not declared	Not declared
`datasets_list`List all datasets	List all datasets	Not declared	Not declared	Not declared	Not declared
`datasets_update`Update a dataset	Update a dataset	Not declared	Not declared	Not declared	Not declared
`judges_compare`Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.	Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.	Not declared	Not declared	Not declared	Not declared
`judges_replay`Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.	Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.	Not declared	Not declared	Not declared	Not declared
`metric_groups_create`Create a metric group	Create a metric group	Not declared	Not declared	Not declared	Not declared
`metric_groups_delete`Delete a metric group	Delete a metric group	Not declared	Not declared	Not declared	Not declared
`metric_groups_get`Get a metric group by ID	Get a metric group by ID	Not declared	Not declared	Not declared	Not declared
`metric_groups_list`List all metric groups	List all metric groups	Not declared	Not declared	Not declared	Not declared
`metric_groups_update`Update a metric group	Update a metric group	Not declared	Not declared	Not declared	Not declared
`metric_versions_dismiss`Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.	Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published version, publish a different one as current instead.	Not declared	Not declared	Not declared	Not declared
`metric_versions_list`List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.	List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps.	Not declared	Not declared	Not declared	Not declared
`metric_versions_publish`Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.	Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_bands back onto the metric so the judge grades against it.	Not declared	Not declared	Not declared	Not declared
`metrics_create`Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.	Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.	Not declared	Not declared	Not declared	Not declared
`metrics_delete`Delete a metric	Delete a metric	Not declared	Not declared	Not declared	Not declared
`metrics_get`Get a metric by ID	Get a metric by ID	Not declared	Not declared	Not declared	Not declared
`metrics_list`List all metrics	List all metrics	Not declared	Not declared	Not declared	Not declared
`metrics_suggest_variants`Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.	Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire via ActiveSupport::Notifications under completion_kit.judge_suggestion.generated.	Not declared	Not declared	Not declared	Not declared
`metrics_update`Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.	Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals/list_overlap/numeric_equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds/numeric_bounds); valid_json takes no extra keys. target_path is required when target is json_path; a target_path sent with no target sets target to json_path, and one sent with any other target is refused. target_path walks array indices, as in items.0.name; any path may start at the root with $., as in $.items.0.name. list_overlap grades a list-valued field and stores a fraction rather than a verdict: score_by is recall, precision, f1 or jaccard (recall by default), and min is the fraction that counts as a pass (1, a perfect score, by default). numeric_bounds and numeric_equals read a number, ignoring the thousands commas and currency symbol a figure is written with; a comma counts only as a thousands separator, with three digits after it and no decimal point before it, so 1,5 is not a number; numeric_equals takes a tolerance, either a plain number for a fixed amount or a percentage string such as 2% for a share of the expected value. A check with no case_sensitive matches case-insensitively, regex and list_overlap included; send case_sensitive: true to match case exactly. Before 0.29.0 a regex check read a missing case_sensitive as case-sensitive. Set compare_to:"expected" on contains, not_contains, equals, json_path_equals, list_overlap, numeric_equals to grade against each row's own expected_output (ground truth) instead of a constant; drop value (or expected, for json_path_equals) and add expected_path to dig into the answer key when it is JSON.	Not declared	Not declared	Not declared	Not declared
`promptfoo_import`Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.	Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary of what mapped and what was skipped and why; nothing is dropped silently.	Not declared	Not declared	Not declared	Not declared
`prompts_create`Create a prompt	Create a prompt	Not declared	Not declared	Not declared	Not declared
`prompts_delete`Delete a prompt	Delete a prompt	Not declared	Not declared	Not declared	Not declared
`prompts_get`Get a prompt by ID	Get a prompt by ID	Not declared	Not declared	Not declared	Not declared
`prompts_list`List all prompts	List all prompts	Not declared	Not declared	Not declared	Not declared
`prompts_publish`Publish a prompt version, making it the current version	Publish a prompt version, making it the current version	Not declared	Not declared	Not declared	Not declared
`prompts_suggest_improvement`Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).	Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as a Suggestion. Requires a run that has a prompt (not a scoring-only run).	Not declared	Not declared	Not declared	Not declared
`prompts_update`Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing. Promote it with prompts_publish, so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.	Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing. Promote it with prompts_publish, so an agent's edits don't go live without a gate. If it has no runs, it is updated in place.	Not declared	Not declared	Not declared	Not declared
`provider_credentials_create`Create a provider credential	Create a provider credential	Not declared	Not declared	Not declared	Not declared
`provider_credentials_delete`Delete a provider credential	Delete a provider credential	Not declared	Not declared	Not declared	Not declared
`provider_credentials_get`Get a provider credential by ID (API key is not exposed)	Get a provider credential by ID (API key is not exposed)	Not declared	Not declared	Not declared	Not declared
`provider_credentials_list`List all provider credentials (API keys are not exposed)	List all provider credentials (API keys are not exposed)	Not declared	Not declared	Not declared	Not declared
`provider_credentials_update`Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint.	Update a provider credential. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint.	Not declared	Not declared	Not declared	Not declared
`responses_get`Get a specific response	Get a specific response	Not declared	Not declared	Not declared	Not declared
`responses_list`List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.	List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "score_asc" to read the worst rows first. For per-metric averages of the whole run use runs_get instead of aggregating here.	Not declared	Not declared	Not declared	Not declared
`runs_create`Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.	Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones.	Not declared	Not declared	Not declared	Not declared
`runs_delete`Delete a run	Delete a run	Not declared	Not declared	Not declared	Not declared
`runs_generate`Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.	Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise copies the graded dataset column and grades it.	Not declared	Not declared	Not declared	Not declared
`runs_get`Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.	Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to find the metric dragging a prompt down without listing responses.	Not declared	Not declared	Not declared	Not declared
`runs_list`List all runs	List all runs	Not declared	Not declared	Not declared	Not declared
`runs_regrade`Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.	Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an already-generated run.	Not declared	Not declared	Not declared	Not declared
`runs_rerun`Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.	Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean run instead of mixing versions.	Not declared	Not declared	Not declared	Not declared
`runs_retry_failures`Re-run only the failed responses of a run, optionally limited to specific response ids via "only".	Re-run only the failed responses of a run, optionally limited to specific response ids via "only".	Not declared	Not declared	Not declared	Not declared
`runs_update`Update a run	Update a run	Not declared	Not declared	Not declared	Not declared
`tags_create`Create a tag. Color is auto-assigned.	Create a tag. Color is auto-assigned.	Not declared	Not declared	Not declared	Not declared
`tags_delete`Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.	Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.	Not declared	Not declared	Not declared	Not declared
`tags_get`Get a tag by ID	Get a tag by ID	Not declared	Not declared	Not declared	Not declared
`tags_list`List all tags	List all tags	Not declared	Not declared	Not declared	Not declared
`tags_update`Rename a tag.	Rename a tag.	Not declared	Not declared	Not declared	Not declared
`usage_get`Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.	Get this organization's plan usage and limits for the current billing period: runs and prompt fetches used, their limits, how many remain, and when the period resets. Call this to pre-check quota before starting runs. Runs are hard-blocked once the run limit is reached (with a small grace band), so a run over the limit will fail with run_limit_reached.	Not declared	Not declared	Not declared	Not declared

Details

Registry status: Active
Published: Jul 18, 2026
Updated: Jul 18, 2026
Website: https://completionkit.com/⁠ (opens in a new tab)
Repository: https://github.com/homemade-software-inc/completion-kit⁠ (opens in a new tab)
Categories (suggested): Data and analytics AI modelsSuggested categories are assigned automatically from the registry text and may be wrong. Verified operators can correct them.
GDPR: Not declared

Are you the operator? Claim this listing