IA-QA Eval & RAG — 38 tools

io.github.JcJamet/ia-qa-evalVersion 1.0.1

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

Status

Responds112 msNov 14, 2026, 14:39 UTCAnswered our handshake without sign-in.

Registry data may be out of date

We could not refresh it, so this page shows the last copy.

Last checked: Nov 14, 2026, 14:39 UTC
Latency: 112 ms
Remotes responding: 1 of 1
Sign-in: No sign-in
Hosting: Hosted outside the EU (detected)
Registry data as of: Nov 14, 2026, 21:12 UTC

Data from the official MCP Registry, checked by run.zero. How we check

Connect

Streamable HTTP

https://www.ia-qa.com/mcp/evalResponds112 msNov 14, 2026, 14:39 UTC

No sign-in

Claude Code

claude mcp add --transport http ia-qa-eval https://www.ia-qa.com/mcp/eval

Claude Desktop and claude.ai

1
2
3

Settings > Connectors > Add custom connector
Name: ia-qa-eval
Remote MCP server URL: https://www.ia-qa.com/mcp/eval

Cursor

{
  "mcpServers": {
    "ia-qa-eval": {
      "url": "https://www.ia-qa.com/mcp/eval"
    }
  }
}

VS Code

code --add-mcp '{"name":"ia-qa-eval","type":"http","url":"https://www.ia-qa.com/mcp/eval"}'

Built your own server? run.zero hosts MCP servers on European infrastructure.

Host your own MCP server on run.zero

Tools

39 tools

Tools of IA-QA Eval & RAG — 38 tools
	Description		Read-only	Idempotent	Open world
`ab_test_report`Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.	Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: `winner` names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.	No	Yes	Yes	No
`analyze_responses`Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.	Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pairwise cosine agreement, the most-representative output, and the outlier. With a `reference` (ground truth): also ranks every output by closeness (token cosine + ROUGE-L composite) and names the closest. No LLM, no key, runs locally. The reference mode is gate-able in CI; the consensus/outlier mode is a signal (no reference to fail against). You bring the outputs (2+). A 2-way head-to-head with structural JSON diff is compare_responses.	No	Yes	Yes	No
`bm25_score`Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.	Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, OpenSearch, and Weaviate hybrid search. Returns ranked results with normalized scores.	No	Yes	Yes	No
`build_rag_prompt`Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.	Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, system instruction injection, and source attribution.	No	Yes	Yes	No
`compare_models`Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.	Put 2-5 AI models side by side: price per 1k tokens, context window, multimodal, reasoning, provider, pricing page. Facts only: names the cheapest and the largest context, never a "best model" — run your cases on each to choose.	No	Yes	Yes	No
`compare_responses`Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.	Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, length/structure deltas, JSON diff) and a verdict. If a `reference` (ground truth) is given, scores each output against it and picks the closer one. If `model` + `api_key` are given, an LLM judge also picks a qualitative winner for the task. No re-execution — you bring the outputs.	No	Yes	Yes	No
`consistency_check`Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.	Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, deterministic, no API key needed. Limitations: relies on surface-level word matching — "Paris is the capital of France" vs "Paris is the French capital" may score low despite semantic equivalence. True semantic consistency: run_semantic_tests in embedding mode. Essential for determinism testing.	No	Yes	Yes	No
`context_window_check`Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.	Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncation risk.	No	Yes	Yes	No
`conversation_analyze`Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.	Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential for chatbot QA.	No	Yes	Yes	No
`cot_analyzer`Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.	Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning depth. Useful for o1/o3/DeepSeek-R1 evaluation.	No	Yes	Yes	No
`count_tokens`Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.	Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Shows whether a text fits the context window of a model and what it costs. Returns token estimate, character count, and word count.	No	Yes	Yes	No
`embedding_similarity`Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.	Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: true vector embeddings are run_semantic_tests with mode="embeddings" and an OpenAI key. Supports single pair or batch mode with pipe-separated pairs. Useful for RAG retrieval testing, semantic search evaluation, and text deduplication.	No	Yes	Yes	No
`estimate_llm_cost`Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.	Estimate the API cost in USD for a given model and token counts, at the price read daily from the provider's own pricing page (prices_as_of, price_status). Any model from list_llm_models.	No	Yes	Yes	No
`extract_json_from_text`Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.	Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks and inline JSON.	No	Yes	Yes	No
`find_tool`Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.	Search available MCP tools by keyword or category before calling them. Returns names, descriptions and, on request, inputSchemas. Queries are logged 90 days, screened for credentials and personal data. Categories: data, dev, encoding, eval, integrations, llm, meta, qa, rag, security, text, web.	No	Yes	Yes	No
`generate_ci_workflow`Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.	Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA: "eval_contract" runs a .ia-eval.yaml through ia-qa-com/eval-action@v1 (needs a provider key as a repo secret), "cli_checks" runs deterministic tools via npx @ia-qa/cli and fails the build on their verdict, "selector_drift" gates E2E selectors with @ia-qa/self-healing. Deterministic template, no LLM. Returns the YAML, the secrets to create and the next steps. Pair with generate_eval_yaml for the contract.	No	Yes	Yes	No
`generate_eval_yaml`Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.	Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, no API key needed). Returns ready-to-run YAML for the LLM Test Runner (run_eval_contract). Picks appropriate evaluators (cosine_similarity, contains_check, hallucination_check, etc.) based on the task type.	No	Yes	No	Yes
`get_testing_guidelines`Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.	Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for a given topic. Call without a topic to list all available topics. Topics: llm-unit-testing, rag-pipeline, prompt-stability, prompt-ab-testing, embedding-quality, eval-framework, semantic-testing, auto-testing, security, api-testing, ci-cd, multimodal, llm-data-security, agent-observability, pro-tips, learning-paths, golden-dataset, selector-drift, qa-recipes, playbooks. Not sure where to start testing an LLM, RAG pipeline or agent? Call without a topic (or with "start-here"): it maps what you are testing to the tools to call and the output field to gate CI on. A plain question such as "how do I test my RAG" also resolves to the right topic.	No	Yes	Yes	No
`hallucination_check`Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.	Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. Each answer sentence is aligned to its best-matching source sentence, so a number only counts as support when it sits on the SAME statement ("founded in 1998" is not grounded by "sold 1998 units"), and a negation or antonym flip against that sentence returns verdict "contradicted" — the corrupted-fact hallucination that reuses source vocabulary. Limitations: still lexical — it cannot follow a paraphrase, a synonym, or multi-sentence reasoning, so a "well_grounded" verdict means "nothing lexical found", never "verified true". Entailment: run_semantic_tests (NLI/embedding) or a calibrated judge.	No	Yes	Yes	No
`list_llm_models`List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.	List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") or return the full catalog. Use this to discover which models are available before calling an LLM API, or to compare providers.	No	Yes	Yes	No
`llm_format_check`Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.	Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for structured output testing.	No	Yes	Yes	No
`llm_generate`Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.	Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b, qwen/qwen3.8-27b (default "openai/gpt-oss-20b"). HuggingFace models are not served on the server key. A reasoning model that spends max_tokens thinking is retried once at 4096. Rate-limited.	No	Yes	No	Yes
`llm_json_schema_check`Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.	Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-calling and structured output testing.	No	Yes	Yes	No
`llm_output_validator`Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.	Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns. Essential for QA testing LLM-powered features.	No	Yes	Yes	No
`metamorphic_check`Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.	Reference-free stability check: does an assistant's answer stay the same when the QUESTION is transformed (case, typo, paraphrase, reorder, translation)? You bring the outputs; no model is called (tfidf mode is deterministic and free). Default thresholds: case .95, typo .90, paraphrase .80, reorder .80, translation .75 (embeddings only), specialization .60 (advisory, never gated). Returns PASS / FAIL / INVALID — INVALID when the BASE answer is a refusal or too short, so a model that refuses everything never scores 1.0. Pair with run_semantic_tests for correctness.	No	Yes	Yes	No
`model_info`Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.	Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature list. Covers 30+ models from OpenAI, Anthropic, Google, DeepSeek, Meta, Mistral, Cohere, xAI.	No	Yes	Yes	No
`needle_haystack_generate`Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.	Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retrieval accuracy. Returns the full haystack, the question to ask, and metadata. No API key needed.	No	Yes	No	No
`prompt_test_suite`Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.	Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as input for manual or automated LLM evaluation.	No	Yes	Yes	No
`rag_relevance_rank`Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.	Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.	No	Yes	Yes	No
`rate_tool`Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })	Give honest usage feedback on an IA-QA MCP tool. Give a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high (4-5) only if it genuinely solved your need. Maintainers read ratings, one vote per agent and tool a month: comments and per-tool scores are not published, only the overall average and vote count. Comments are screened: one carrying a credential is not stored, personal data becomes placeholders — describe what happened, not what a model should do. Skip routine successes: signal, not praise. Example: rate_tool({ tool_name: "format_json", score: 2, comment: "Tried to pretty-print a JSON5 file, it rejected trailing commas — not usable for my case." })	Yes	No	No	No
`rerank_evaluate`Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).	Evaluate RAG retrieval: rank passages against a query, compute Precision@k / Recall@k and a PASS/FAIL verdict from your relevance labels (INCONCLUSIVE without them). Modes: (1) give every passage the `score` YOUR reranker produced — deterministic, offline, the mode to gate CI on; (2) no scores, no key — a local BM25 keyword baseline, never a judgement of a neural reranker; (3) api_key for an NVIDIA account that still has reranking (NVIDIA retired the hosted endpoints on 2026-05-18).	No	Yes	No	Yes
`response_quality_score`Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.	Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actually measured — or total_score null with the reason when none is given, because relevance and correctness cannot be read off word overlap (a correct one-word answer shares no words with its question). Meaning against a reference answer: run_semantic_tests. Also returns unscored signals: question-term overlap, average sentence length, markdown.	No	Yes	Yes	No
`run_eval_contract`Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.	Run an IA-EVAL test suite (.ia-eval.yaml contract): call the model per scenario; return passed (true, false, null if undecided) and a verdict (PASS, FAIL, PARTIAL). Expectations are assertions that fail a scenario whatever its score; evaluators give the score. Hosted: pass inline_contract. validate_only calls no model.	No	Yes	No	Yes
`run_semantic_tests`Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.	Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embeddings (OpenAI text-embedding-3-small, BYOK, true semantic similarity). Returns per-case PASS/FAIL verdicts and an overall verdict. CI-ready: pipe the JSON verdict field to gate a build.	No	Yes	Yes	No
`run_vlm_test_suite`Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, `overall` PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.	Test a vision model (OpenAI, Anthropic or Gemini) on one image: each test case asks a question and asserts on the answer (contains, not_contains, json_format, min_length, max_length, semantic_contains = TF-IDF cosine ≥ 0.4). Returns per-case PASS/FAIL, the pass rate, `overall` PASS/WARNING/FAIL against threshold (N/A when no assertion could run), and latency. BYOK: your key for the model's provider; the image is fetched only from a public address.	No	Yes	No	Yes
`similarity_score`Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.	Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key needed. Ideal for LLM eval (expected vs actual), RAG quality checks, and NLG benchmarking. Supports batch mode.	No	Yes	Yes	No
`skill_interview`Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.	Help a person write a SKILL.md they can test. No answers: the questions (why + example), written for the person; invented answers make the result meaningless. With answers: remarks per answer (no model), the SKILL.md, scenarios for test_skill, a starter .ia-eval.yaml.	No	Yes	Yes	No
`test_skill`Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.	Test a SKILL.md. No model, every call: lint (gate on lint.valid) and review (description lacks what/when, words shared with other_skills, leftovers, no done-check, missing files; why + fix each). Unless lint_only, a model picks a skill per request from names+descriptions: trigger_accuracy PASS ≥80, FIX ≥50, else BLOCK; your scenarios = gate, generated = signal; unanswered = INCONCLUSIVE. Server key 10 runs/day.	No	Yes	No	Yes
`token_budget_calculator`Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).	Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds the model's context window (any model from list_llm_models).	No	Yes	Yes	No

Details

Registry status: Active
Published: Oct 26, 2026
Updated: Oct 26, 2026
Website: Not given
Repository: https://github.com/ia-qa-com/mcp⁠ (opens in a new tab)
Categories (suggested): AI modelsSuggested categories are assigned automatically from the registry text and may be wrong. Verified operators can correct them.
GDPR: Not declared

Are you the operator? Claim this listing